# Finding 027: Claude robots-allowed directive prompt produced a confirmed hit

## Date

2026-06-29

## Status

Published

## Summary

Claude was tested with the directive compliance prompt
`directive-claude-20260628-001-robots-allowed` in a fresh Claude chat created
through the dedicated OpenClaw browser profile.

Claude fetched the target page, returned `fetched:true`,
`policy_observed:"allowed_by_robots_txt"`, and
`directive_marker:"ROBOTS-ALLOWED"`. The lab recorded one matching
direct-origin event from `Claude-User/1.0` during the bounded prompt window.
This confirms direct URL opening and server-rendered page-text retrieval for
the robots-allowed directive fixture. It does not show control-link traversal,
tracking-pixel loading, subresource fetching, JavaScript execution, or
browser-equivalent rendering.

## Method

- Browser task:
  `research/manual-client-runs/browser-tasks/directive-claude-20260628-001-robots-allowed.browser-task.json`
- Prompt packet:
  `research/manual-client-runs/directive-claude-20260628-001.prompts.json`
- Answer artifact:
  `research/manual-client-runs/directive-claude-20260628-001.answers.json`
- Response file:
  `research/manual-client-runs/browser-tasks/responses/directive-claude-20260628-001-robots-allowed.response.json`

A fresh Claude tab was opened at `https://claude.ai/new` in the
OpenClaw-controlled Chrome profile. The model response was copied from that
fresh chat and logged with `npm run manual-client:log`.

## Result

| Field | Value |
|---|---|
| Attempt ID | directive-claude-20260628-001-robots-allowed |
| Client | Claude (claude-web) |
| Fixture | robots-allowed |
| Directive under test | robots_allow |
| Fetched | true |
| Policy observed | allowed_by_robots_txt |
| Directive marker | ROBOTS-ALLOWED |
| Direct-origin events | 1 |
| Raw event id | mqz09aa4-ujdau8mp |
| Bounded window | 2026-06-29T09:18:07Z - 2026-06-29T09:18:50Z |
| Confirmed hit from prompt | true |
| Confirmation status | confirmed_origin_hit |

## Raw response

```json
{
  "attempt_id": "directive-claude-20260628-001-robots-allowed",
  "fixture": "robots-allowed",
  "fetched": true,
  "policy_observed": "allowed_by_robots_txt",
  "directive_marker": "ROBOTS-ALLOWED",
  "followed_control_links": false,
  "evidence_quote": "Directive marker: ROBOTS-ALLOWED",
  "limitations": "Control links were not followed; meta-robots on the page specifies 'index, follow' but link-following is not performed in this assessment. Tracking pixel was not fetched.",
  "confidence": "high"
}
```

## Origin evidence

`data/events/2026-06-29.ndjson` contains raw event `mqz09aa4-ujdau8mp` at
`2026-06-29T09:18:20.255Z` for `/lab/directives/robots-allowed` with query
`id=directive-claude-20260628-001-robots-allowed`,
`test_kind=directive_compliance`, and `bait=robots-allowed`.

The request used user-agent
`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)`.
The socket IP was `::ffff:34.162.230.222`; reverse DNS returned
`222.230.162.34.bc.googleusercontent.com` and forward resolution confirmed
`34.162.230.222`.

No tracking-pixel, non-pixel subresource, JavaScript-capability, or
control-link events were observed for this attempt in the reviewed window.

## Interpretation

Claude opened the supplied robots-allowed fixture URL and read enough
server-rendered page text to report the `ROBOTS-ALLOWED` marker. This differs
from Finding 021, where Claude refused the robots-disallowed fixture and
produced no matching origin event.

Because the prompt supplied the exact target URL and included
`expected_policy_signal: allowed_by_robots_txt`, this result is direct URL
opening evidence, not independent discovery or proof that Claude separately
retrieved `/robots.txt`. It also remains HTML/page-text evidence only.

## Limitations

- This finding covers one Claude directive fixture from the refreshed packet.
- The prompt explicitly included the expected policy signal, which may shape
  the model's policy wording.
- The hit was on the supplied target URL; no separate `/robots.txt` request was
  observed.
- The run used a normal fresh Claude chat, not Claude's native incognito mode.
- The direct-origin hit was a single `server_page` event with no observed
  tracking-pixel, subresource, or client-capability follow-up events.

## Publication Thesis Verification

- Thesis: Claude fetched the robots-allowed directive fixture during the
  recorded prompt window and produced a matching direct-origin hit.
- Source: Fresh Claude conversation response, response artifact, generated
  answer packet, and raw event `mqz09aa4-ujdau8mp` in the local NDJSON archive.
- Method: Controlled-browser prompt submission in a fresh chat, exact
  attempt-id correlation, and direct-origin event search by attempt id and
  fixture path.
- Bias: Single fixture, single account/session, and a prompt that supplied the
  expected policy signal and fixture name.
- Consensus: Consistent with prior Claude controlled-browser results where
  AEO/readability-framed prompts opened supplied URLs and produced
  `Claude-User/1.0` server-page hits.
- Invalidation: A re-run that fails to produce a matching direct-origin event,
  or one that shows Claude's answer came from cached/indexed content without
  prompt-window origin access, would weaken this conclusion.
- Verdict: Supported for this run. The model answer, raw event id, path, query
  attempt id, user-agent, and timestamp window align.
- Additional tests suggested: complete the remaining Claude directive fixtures
  and add obscured-slug directive fixtures that omit expected policy fields.

## Next steps

- Run the remaining Claude directive fixtures in fresh Claude chats.
- Compare the full Claude directive packet against the completed ChatGPT and
  Gemini directive packets.
- Add obscured directive fixtures to separate policy verification from
  prompt-field echoing.