How do major AI assistants retrieve, read, and interact with web pages? Based on 95 controlled-browser tests with direct-origin server evidence — not just what the models claim.
Anthropic · claude.ai
Claude-User/1.0
Google · gemini.google.com
Google
OpenAI · chatgpt.com
ChatGPT-User/1.0
Perplexity AI · perplexity.ai
— (no hits)
Microsoft · copilot.microsoft.com
— (no hits)
Each cell is backed by controlled lab tests with direct-origin server evidence.
| Capability | Claude | Gemini | ChatGPT | Perplexity | Copilot |
|---|---|---|---|---|---|
| Fetches target URL | ✓ | ✓ | ◐ 89% | ✕ | ✕ |
| Reads visible HTML text | ✓ | ✓ | ✓ | — | — |
| Reads JS-rendered content | ✕ | ✕ | ✕ | — | — |
| Reads image alt text | ✓ | ✓ | ✓ | — | — |
| Reads image pixels | ✕ | ✕ | ✕ | — | — |
| Follows visible links (depth-1) | ✓ | ◐ claimed | ✕ | — | — |
| Exposes hidden/comment hrefs | ✕ | ✕ | ✕ | — | — |
| Fetches subresources (CSS, JS, fonts) | ✕ | ✕ | ✕ | — | — |
| Executes JavaScript | ✕ | ✕ | ✕ | — | — |
| Loads tracking pixels | ✕ | ✕ | ✕ | — | — |
| Respects robots.txt | ✕ fetched | ✕ fetched | ✕ fetched | — | — |
| Respects meta noindex | ✕ fetched | ✕ fetched | ✕ fetched | — | — |
| Reads consent banners | ✓ | ✓ | ✓ | — | — |
| Interacts with consent | ✕ | ✕ | ✕ | — | — |
| Finds sitemap-only pages | ✓ | ✓ | ✓ | — | — |
| Finds robots-only pages | ✓ | ✓ | ✓ | — | — |
"Fetched" means the AI retrieved the page despite the directive — none of the tested clients respected robots.txt or meta noindex. "— (not tested)" means the client never successfully fetched any URL, so downstream capabilities could not be measured.
Claude, Gemini, and ChatGPT reliably fetch target URLs. Perplexity and Copilot/Bing reliably cannot or do not. This split was consistent across all 95 tests.
Even when AI clients fetch successfully, none execute JavaScript, load tracking pixels, fetch subresources, or perform browser-equivalent rendering. Retrieval is page-text/HTML only.
No AI client respected or referenced robots.txt or meta noindex directives when fetching target URLs. Claude and Gemini fetched noindex pages without acknowledging the directive.
Gemini depends on Google Search index availability rather than direct URL fetching. Pages not in the index return NOT_IN_SEARCH_INDEX errors, even for robots-allowed URLs.
Claude refused measurement-framed prompts but successfully fetched the same URLs when reframed as site-owner AEO/readability work. This is a prompt-framing dependency, not a retrieval limitation.
ChatGPT's 2/19 no-hits were a URL-safety guardrail (p08) and a fetch-depth limitation (p17) — not systematic retrieval failures. All other 17 tests produced confirmed hits.
Each test was run from a prepared browser-task artifact in a fresh AI-client chat. The lab server independently logged all incoming requests with full headers, timing, IP, DNS, and user-agent. After each run, model answers were correlated with direct-origin events by prompt code, source prompt ID, and bounded timestamp windows.