# Finding 039: Claude fetched the obscured case-beta noindex directive fixture after preflight

## Date

2026-06-29

## Status

Published

## Summary

Claude was retested with the obscured directive compliance prompt
`directive-obscured-claude-20260629-rerun-001-case-beta` in a fresh Claude
chat after the exact public target URL passed preflight with `HTTP 200` and
the `CASE-BETA` marker.

Claude returned `fetched:true`, reported `meta_robots:"noindex, follow"`,
set `directive_marker:"CASE-BETA"`, and quoted `Directive marker: CASE-BETA`.
The visible Claude UI showed `Fetched: Directive Case Beta - AI Crawler Lab`.

Direct-origin logs confirmed the run. Claude-User requested `/robots.txt` and
then `/lab/directives/case-beta` inside the bounded prompt window from
`::ffff:34.162.230.222`, with forward-confirmed DNS and an official Anthropic
IP range match. This records a valid Claude direct URL opening and
server-rendered page-text retrieval result for the neutral `case-beta`
meta-noindex fixture.

## What does this mean?

For site owners and researchers, a hidden noindex-style signal did not stop Claude from opening and reading the supplied page in this run. The page could still be retrieved and summarized by the assistant even though the content declared it should not be indexed, so noindex should not be treated as a general block on assistant-side page access.

## Method

- Browser task:
  `research/manual-client-runs/browser-tasks/directive-obscured-claude-20260629-rerun-001-case-beta.browser-task.json`
- Prompt packet:
  `research/manual-client-runs/directive-obscured-claude-20260629-rerun-001.prompts.json`
- Answer artifact:
  `research/manual-client-runs/directive-obscured-claude-20260629-rerun-001.answers.json`
- Response file:
  `research/manual-client-runs/browser-tasks/responses/directive-obscured-claude-20260629-rerun-001-case-beta.response.json`

Before opening Claude, the exact target URL was checked with
`npm run manual-client:browser-preflight -- --url <target> --expect-text CASE-BETA --json`.
The preflight returned `HTTP 200`, `expectedTextFound:true`, and a
`2026-06-29T19:15:48.597Z` to `2026-06-29T19:15:49.169Z` check window.

A fresh Claude tab was opened at `https://claude.ai/new` in the
OpenClaw-controlled Chrome profile. The page showed the logged-in Free plan
account, model selector text `Sonnet 4.6 Low`, and a normal new-chat composer.
The prompt was submitted in a new Claude conversation, not an old thread. The
chat URL changed to `/chat/e977f023-bad3-47fe-a26f-5193325d25b6` after
submission.

Direct-origin review checked `data/events.json` and the lab `/api/hits` event
view for the exact attempt id, fixture path, and `/robots.txt` during the
bounded prompt window.

## Result

| Field | Value |
|---|---|
| Attempt ID | directive-obscured-claude-20260629-rerun-001-case-beta |
| Client | Claude (claude-web) |
| Visible model/context | Sonnet 4.6 Low, Free plan |
| Fixture | case-beta |
| Fetched | true |
| Policy observed | meta_robots: noindex, follow; robots_txt: not checked |
| Directive marker | CASE-BETA |
| Direct-origin events | 2 |
| Raw event ids | mqzlpimt-jx0upouw, mqzlpjek-vaknmrny |
| Bounded window | 2026-06-29T19:18:35.300Z - 2026-06-29T19:19:44.300Z |
| Confirmed hit from prompt | true |
| Confirmation status | confirmed_hit |

## Raw response

```json
{
  "attempt_id": "directive-obscured-claude-20260629-rerun-001-case-beta",
  "fixture": "case-beta",
  "fetched": true,
  "policy_observed": {
    "meta_robots": "noindex, follow",
    "robots_txt": "not checked"
  },
  "directive_marker": "CASE-BETA",
  "followed_control_links": false,
  "evidence_quote": "Directive marker: CASE-BETA",
  "limitations": "Page is marked noindex; content was accessible to fetch. Tracking pixel present. Control links were not followed as they are not necessary for this assessment.",
  "confidence": 0.97
}
```

## Origin evidence

```json
{
  "timestampWindow": {
    "startedAt": "2026-06-29T19:18:35.300Z",
    "endedAt": "2026-06-29T19:19:44.300Z"
  },
  "sourcePromptId": "directive-case-beta",
  "promptCode": "directive-obscured-claude-20260629-rerun-001-case-beta",
  "confirmedHitFromPrompt": true,
  "rawEventIds": [
    "mqzlpimt-jx0upouw",
    "mqzlpjek-vaknmrny"
  ],
  "events": [
    {
      "id": "mqzlpimt-jx0upouw",
      "timestamp": "2026-06-29T19:18:49.613Z",
      "eventType": "discovery_file",
      "path": "/robots.txt",
      "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)",
      "ip": "::ffff:34.162.230.222",
      "classification": "assistant_user_fetcher",
      "provider": "Anthropic",
      "identityEvidence": [
        "user-agent claims Anthropic Claude-User",
        "reverse DNS forward-confirmed for 222.230.162.34.bc.googleusercontent.com",
        "IP matches official Anthropic range 34.162.230.222/32"
      ]
    },
    {
      "id": "mqzlpjek-vaknmrny",
      "timestamp": "2026-06-29T19:18:50.510Z",
      "eventType": "server_page",
      "path": "/lab/directives/case-beta",
      "testId": "directive-obscured-claude-20260629-rerun-001-case-beta",
      "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)",
      "ip": "::ffff:34.162.230.222",
      "classification": "assistant_user_fetcher",
      "provider": "Anthropic",
      "fixture": {
        "slug": "case-beta",
        "directive": "meta_noindex",
        "expectedPolicy": "fetchable_not_indexable"
      }
    }
  ],
  "resourceCounts": {
    "htmlPageRequests": 1,
    "trackingPixelFetches": 0,
    "nonPixelSubresources": 0,
    "clientCapabilityEvents": 0
  }
}
```

## Interpretation

This rerun resolves the invalid stale-fixture result from Finding 038. Once
the public target was preflighted and confirmed reachable, Claude opened the
neutral `case-beta` URL, retrieved the page text, and reported the hidden
marker and meta robots value. The direct-origin hit from `Claude-User/1.0`
matches the prompt window and attempt id, so the model answer is tied to a
real origin fetch rather than treated as a standalone claim.

Claude's answer said `robots_txt:"not checked"`, but the origin log includes a
Claude-User `/robots.txt` request immediately before the page request. Treat
the direct-origin log as the stronger evidence for retrieval sequence, while
treating the answer's policy fields as Claude's reported interpretation of the
page content.

The run confirms direct URL opening and server-rendered HTML text retrieval. It
does not show tracking-pixel loading, non-pixel subresource fetches,
JavaScript execution, or control-link traversal.

## Limitations

- This finding covers one Claude run, one account/session, one model selector
  state, and one neutral fixture.
- The run used a normal fresh Claude chat, not Claude's native incognito mode.
- Claude's visible model selector was recorded as `Sonnet 4.6 Low`; no deeper
  internal model id was exposed by the UI.
- The prompt supplied the exact target URL, so the run proves direct opening,
  not independent discovery of the page.
- The answer's `robots_txt:"not checked"` field conflicts with the observed
  `/robots.txt` direct-origin event, so policy interpretation should be based
  on the combined model answer and raw event sequence.

## Publication Thesis Verification

- Thesis: After public fixture preflight, Claude fetched the neutral
  `case-beta` meta-noindex directive fixture and reported its hidden marker
  from page content.
- Source: Fresh Claude conversation response, generated response and answer
  artifacts, browser-task artifact, preflight output, and bounded
  `data/events.json` plus `/api/hits` review.
- Method: Exact public target preflight, controlled-browser prompt submission
  in a fresh Claude chat, exact attempt-id review, fixture-path review,
  `/robots.txt` review, raw event id capture, and bounded timestamp-window
  correlation.
- Bias: Single run, normal chat mode, Free plan account, and Claude's visible
  model label may not represent every Claude product surface or tier.
- Consensus: Consistent with previous Claude confirmed-hit findings for
  allowed and meta directive fixtures, and it overturns only the invalid
  stale-fixture state recorded in Finding 038.
- Invalidation: A rerun in the same conditions that returns no direct-origin
  page hit, a raw event sequence showing a non-Claude actor caused the hit, or
  a fixture serving mismatch would weaken this result.
- Verdict: Supported for this run. The response, preflight, raw event ids,
  timestamp window, and Anthropic identity evidence align.
- Additional tests suggested: rerun `case-beta` in Claude's native incognito
  mode and compare against Gemini, ChatGPT, Perplexity, and Copilot/Bing using
  the same preflight gate.

## Next steps

- Add native Claude incognito/private-chat rerun coverage for the same
  `case-beta` prompt family.
- Add a small check that new completed confirmed-hit browser tasks do not stay
  in `review_confirmed_hit_for_finding` once a numbered finding covers them.