AI Crawler Lab

Research harness for measuring how AI crawlers, search agents, and browser-backed assistants retrieve websites.

Start Experiments

• Observed server-side page
• Crawl depth start
• Reading tests
• Dashboard

Documentation

• Project plan
• AI systems catalog
• Top-five test plans
• ChatGPT reading experiments
• Fingerprint confidence model
• Research methodology
• Signal catalog
• Finding template
• Findings index
• Finding 001: multi-signal classification
• Finding 002: OpenRouter claims need direct lab evidence
• Finding 003: per-model OpenRouter probe attribution
• Finding 004: OpenClaw web fetch reading baseline
• Finding 005: local HTTP directive compliance baseline