# Finding 028: Claude robots-crawl-delay directive prompt produced a confirmed hit

## Date

2026-06-29

## Status

Published

## Summary

Claude was tested with the directive compliance prompt
`directive-claude-20260628-001-robots-crawl-delay` in a fresh Claude chat
created through the dedicated OpenClaw browser profile.

Claude fetched the target page, returned `fetched:true`,
`policy_observed:"fetchable_with_crawl_delay_signal"`, and
`directive_marker:"ROBOTS-CRAWL-DELAY"`. The lab recorded one matching
direct-origin event from `Claude-User/1.0` during the bounded prompt window.
This confirms direct URL opening and server-rendered page-text retrieval for
the robots-crawl-delay fixture. It does not show independent `/robots.txt`
retrieval, crawl-delay timing enforcement, control-link traversal,
tracking-pixel loading, subresource fetching, JavaScript execution, or
browser-equivalent rendering.

## Method

- Browser task:
  `research/manual-client-runs/browser-tasks/directive-claude-20260628-001-robots-crawl-delay.browser-task.json`
- Prompt packet:
  `research/manual-client-runs/directive-claude-20260628-001.prompts.json`
- Answer artifact:
  `research/manual-client-runs/directive-claude-20260628-001.answers.json`
- Response file:
  `research/manual-client-runs/browser-tasks/responses/directive-claude-20260628-001-robots-crawl-delay.response.json`

A fresh Claude tab was opened at `https://claude.ai/new` in the
OpenClaw-controlled Chrome profile. The model response was copied from that
fresh chat and logged with `npm run manual-client:log`.

## Result

| Field | Value |
|---|---|
| Attempt ID | directive-claude-20260628-001-robots-crawl-delay |
| Client | Claude (claude-web) |
| Fixture | robots-crawl-delay |
| Directive under test | robots_crawl_delay |
| Fetched | true |
| Policy observed | fetchable_with_crawl_delay_signal |
| Directive marker | ROBOTS-CRAWL-DELAY |
| Direct-origin events | 1 |
| Raw event id | mqz0s1fi-ll2f2zna |
| Bounded window | 2026-06-29T09:32:39.300Z - 2026-06-29T09:33:55.300Z |
| Confirmed hit from prompt | true |
| Confirmation status | confirmed_hit |

## Raw response

```json
{
  "attempt_id": "directive-claude-20260628-001-robots-crawl-delay",
  "fixture": "robots-crawl-delay",
  "fetched": true,
  "policy_observed": "fetchable_with_crawl_delay_signal",
  "directive_marker": "ROBOTS-CRAWL-DELAY",
  "followed_control_links": false,
  "evidence_quote": "Directive marker: ROBOTS-CRAWL-DELAY",
  "limitations": "Crawl-delay is a robots.txt directive not enforced at the HTTP or HTML level; this client fetched the page without applying any crawl delay. The crawl-delay signal was observed in page content only, not via robots.txt inspection. Control links were not followed as not required by the task.",
  "confidence": 0.92
}
```

## Origin evidence

`data/events.json` contains raw event `mqz0s1fi-ll2f2zna` at
`2026-06-29T09:32:55.274Z` for `/lab/directives/robots-crawl-delay` with query
`id=directive-claude-20260628-001-robots-crawl-delay`,
`test_kind=directive_compliance`, and `bait=robots-crawl-delay`.

The request used user-agent
`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)`.
The socket IP was `::ffff:34.162.230.222`; reverse DNS returned
`222.230.162.34.bc.googleusercontent.com`, forward resolution confirmed
`34.162.230.222`, and the IP matched Anthropic's published
`34.162.230.222/32` range.

No tracking-pixel, non-pixel subresource, JavaScript-capability, `/robots.txt`,
or control-link events were observed for this attempt in the reviewed window.

## Interpretation

Claude opened the supplied robots-crawl-delay fixture URL and read enough
server-rendered page text to report the `ROBOTS-CRAWL-DELAY` marker. This
matches Finding 027's robots-allowed fetch behavior and differs from Finding
021, where Claude refused the robots-disallowed fixture and produced no
matching origin event.

Because the prompt supplied the exact target URL and included
`expected_policy_signal: fetchable_with_crawl_delay_signal`, this result is
direct URL opening evidence, not independent discovery or proof that Claude
separately retrieved `/robots.txt`. Claude's own limitation text also says the
crawl-delay signal was observed in page content only, not through robots.txt
inspection.

## Limitations

- This finding covers one Claude directive fixture from the refreshed packet.
- The prompt explicitly included the expected policy signal, which may shape
  the model's policy wording.
- The hit was on the supplied target URL; no separate `/robots.txt` request was
  observed.
- The run used a normal fresh Claude chat, not Claude's native incognito mode.
- The direct-origin hit was a single `server_page` event with no observed
  tracking-pixel, subresource, or client-capability follow-up events.

## Publication Thesis Verification

- Thesis: Claude fetched the robots-crawl-delay directive fixture during the
  recorded prompt window and produced a matching direct-origin hit.
- Source: Fresh Claude conversation response, response artifact, generated
  answer packet, and raw event `mqz0s1fi-ll2f2zna` in the local event store.
- Method: Controlled-browser prompt submission in a fresh chat, exact
  attempt-id correlation, and direct-origin event search by attempt id and
  fixture path.
- Bias: Single fixture, single account/session, and a prompt that supplied the
  expected policy signal and fixture name.
- Consensus: Consistent with Claude's robots-allowed directive result and
  prior Claude AEO/readability-framed prompt runs that opened supplied URLs
  and produced `Claude-User/1.0` server-page hits.
- Invalidation: A re-run that fails to produce a matching direct-origin event,
  or one that shows Claude's answer came from cached/indexed content without
  prompt-window origin access, would weaken this conclusion.
- Verdict: Supported for this run. The model answer, raw event id, path, query
  attempt id, user-agent, official-range evidence, and timestamp window align.
- Additional tests suggested: complete the remaining Claude meta directive
  fixtures and add obscured-slug directive fixtures that omit expected policy
  fields.

## Next steps

- Run the remaining Claude meta directive fixtures in fresh Claude chats.
- Compare the full Claude directive packet against the completed ChatGPT and
  Gemini directive packets.
- Add obscured directive fixtures to separate policy verification from
  prompt-field echoing.