# Finding 058: ChatGPT lost retrieval on the neutral prompt while Claude kept the same extraction pattern ## Date 2026-07-01 ## Status Published ## Summary Findings 041, 042, 047, 049, 052, 053, and 057 isolate the ChatGPT and Claude runs across p20, p21, and p22. All three prompts used exact target URLs. p20 visibly announced that page body text, metadata, and JSON-LD disagreed. p21 served the same hidden-signal layout without the visible conflict cue, but still asked directly for visible, meta-description, and JSON-LD fields. p22 reused the p21 target page and asked only for a neutral site-owner page-quality summary. ChatGPT had confirmed target-page hits for p20 and p21, then returned `fetched:false` with no direct-origin hit for p22. In the two confirmed-hit runs, ChatGPT reported only the visible body code and did not expose meta-description or JSON-LD codes. Claude had confirmed target-page hits for all three prompts. In each confirmed-hit run, Claude surfaced visible body text plus meta-description text, and did not expose JSON-LD in the copied answer. ## What does this mean? For site owners and researchers, small changes in how a question is framed can change whether an assistant reaches the page at all, not just which parts of the page it summarizes. In these runs, ChatGPT fetched the page when the prompt explicitly asked about hidden fields but did not fetch the same target under neutral wording. Claude fetched every variant and consistently exposed the meta description alongside visible text while leaving JSON-LD out, so its behavior looks more stable across prompt wording for this fixture. ## Method This comparison reviews: - ChatGPT p20: Finding 041, `manual-client-chatgpt-20260625-001-p20`. - Claude p20: Finding 042, `manual-client-claude-20260625-001-p20`. - Claude p21: Finding 047, `manual-client-claude-20260625-001-p21`. - ChatGPT p21: Finding 049, `manual-client-chatgpt-20260625-001-p21`. - ChatGPT p22: Finding 052, `manual-client-chatgpt-20260625-001-p22`. - Claude p22: Finding 053, `manual-client-claude-20260625-001-p22`. - p20/p21 comparison: Finding 051. - p22 comparison: Finding 057. - Response artifacts: `research/manual-client-runs/browser-tasks/responses/manual-client-chatgpt-20260625-001-p20.response.json`, `research/manual-client-runs/browser-tasks/responses/manual-client-chatgpt-20260625-001-p21.response.json`, `research/manual-client-runs/browser-tasks/responses/manual-client-chatgpt-20260625-001-p22.response.json`, `research/manual-client-runs/browser-tasks/responses/manual-client-claude-20260625-001-p20.response.json`, `research/manual-client-runs/browser-tasks/responses/manual-client-claude-20260625-001-p21.response.json`, and `research/manual-client-runs/browser-tasks/responses/manual-client-claude-20260625-001-p22.response.json`. Each underlying run used a fresh AI-client chat or native private/temporary thread, logged the copied answer with `npm run manual-client:log`, and reviewed direct-origin events by `promptCode`, `sourcePromptId`, `confirmedHitFromPrompt`, `rawEventIds`, and bounded timestamp windows. ## Comparison Matrix | Client | p20 visible conflict | p21 hidden conflict | p22 neutral summary | Transition | |---|---|---|---|---| | ChatGPT | Confirmed target hit; reported `VISIBLE-SILVER-30`; metadata and JSON-LD not exposed. | Confirmed target hit; reported `VISIBLE-PLUM-47`; metadata and JSON-LD not exposed. | `fetched:false`, `pages_opened:0`; clean bounded no-hit for exact attempt and fixture path. | Retrieval changed between directed hidden-field prompts and the neutral prompt; content conclusions stop at p20/p21. | | Claude | Confirmed target hit plus `/robots.txt`; reported `VISIBLE-SILVER-30` and `META-TEAL-55`; JSON-LD unobserved. | Confirmed target hit plus `/robots.txt`; reported `VISIBLE-PLUM-47` and `META-AMBER-16`; JSON-LD unobserved. | Confirmed target hit plus `/robots.txt`; reported `VISIBLE-PLUM-47` and `META-AMBER-16`; JSON-LD unobserved. | Retrieval and extraction pattern stayed stable across visible-conflict, hidden-field, and neutral wording. | ## Interpretation The retrieval transition matters because the p22 ChatGPT no-hit is not a hidden-signal result. ChatGPT's p20 and p21 findings support a visible-text only observation for confirmed target-page hits, but p22 cannot be used to say whether ChatGPT would have used visible text, metadata, or JSON-LD under neutral wording. The origin logs show that it did not reach the page in that attempt. Claude is the stronger cross-prompt comparison. The p20 prompt named the conflict, p21 directly asked for hidden fields without the visible conflict cue, and p22 did not ask for metadata or JSON-LD at all. Claude still fetched the target and surfaced meta-description text in every run, while JSON-LD remained absent from the copied answer. That suggests Claude's extraction view for this fixture includes meta-description text as ordinary extracted content, not only when the prompt explicitly asks for it. The two clients therefore need different next tests. ChatGPT needs retrieval stability reruns that vary neutral wording, task framing, and target URL shape. Claude needs content-surface reruns that isolate meta-description extraction from raw-source or rendered-DOM access, especially before making broader claims about hidden page signals. ## Limitations - This comparison covers one p20, p21, and p22 attempt per client, with the logged accounts, modes, and timestamp windows from the source findings. - All prompts supplied exact target URLs, so these are direct-opening tests, not independent discovery tests. - ChatGPT p22 is a no-hit. It cannot support page-field visibility conclusions. - Claude's repeated JSON-LD absence means JSON-LD did not appear in the copied answers; it does not prove the underlying service cannot inspect JSON-LD in other modes or future runs. - p20 and p21 target pages differ by fixture wording and codes, while p22 reuses the p21 page with a neutral prompt. The comparison is about observed prompt-family transitions, not a fully controlled factorial experiment. ## Publication Thesis Verification - Thesis: Across p20-p22, ChatGPT shifted from confirmed p20/p21 target hits with visible-only answers to a p22 target no-hit, while Claude confirmed target hits in all three runs and repeatedly surfaced visible text plus meta-description text without JSON-LD. - Source: Findings 041, 042, 047, 049, 052, 053, 057, browser-task response artifacts for the six ChatGPT/Claude attempts, and public evidence bundles for the cited findings. - Method: Compare target-hit status, copied answer fields, raw event ids, ancillary `/robots.txt` activity, and timestamp windows across the three prompt families while separating retrieval evidence from answer content. - Bias: The comparison uses single controlled-browser attempts under observed account tiers and client modes. Product routing, account state, retrieval tool availability, and time can change outcomes. - Consensus: Consistent with Findings 051 and 057: ChatGPT was visible-only on p20/p21 confirmed hits and no-hit on p22; Claude exposed visible plus meta-description text on every confirmed p20-p22 hit. - Invalidation: A corrected raw-event review, response metadata mismatch, fixture-serving error, or same-condition rerun that changes the target-hit status or exposes JSON-LD would weaken this comparison. - Verdict: Supported for the published p20-p22 ChatGPT and Claude corpus. The finding should stay scoped to these runs and should drive rerun design rather than broad platform claims. - Additional tests suggested: add ChatGPT neutral-wording reruns that keep the p21 target but vary task framing, and add Claude metadata-isolation reruns that compare meta-description, OpenGraph, canonical, and JSON-LD exposure. ## Next steps - Prepare a small ChatGPT neutral retrieval-stability packet that reuses the hidden structured-data target with two or three neutral prompt framings. - Prepare a Claude metadata-isolation prompt family that separates ordinary meta description text from other head and structured-data fields.