# Research Methodology

## Objective

Measure how AI systems retrieve, search, index, summarize, and traverse websites.

The goal is not perfect attribution on day one. The goal is a repeatable
evidence trail that lets humans and AI systems verify crawler behavior.

## Observation Layers

### 1. Server-Side Page Request

Every experiment page is served by a function, not as a static file. The server
records the request before the client has a chance to execute JavaScript.

Record:

- timestamp
- path and experiment id
- method
- IP address as seen by the platform
- forwarded headers
- user-agent
- accept headers
- fetch metadata headers
- referrer
- language
- raw header subset

### 2. On-Page Analytics Event

Controlled pages should include a first-party, privacy-minimal analytics script
when JavaScript runtime behavior is part of the experiment. The event proves
that the client executed page code; absence of the event is evidence for an
HTML-only or no-JavaScript retrieval path.

Record:

- attempt id and prompt code
- source prompt id and fixture id
- generated page-view id
- JavaScript execution timestamp
- visibility state
- coarse viewport class
- coarse timing

Do not use this layer to collect personal data or identify a human. It exists
to compare server logs, resource beacons, and runtime capability events.

### 3. Tracking Pixel And Subresource Fetches

Experiment pages include multiple resource types:

- image pixel
- CSS file
- script file
- JSON endpoint
- deep links

This distinguishes HTML-only fetchers from rendering browsers and headless
browser agents.

Every standard test should compare server page request, tracking-pixel hit, and
on-page analytics event. If one layer is absent, document that absence instead
of treating it as a failed test.

### 4. Client Capability Events

Human browsers and browser-backed agents can report runtime behavior:

- script loaded
- page visibility
- pointer movement count
- scroll depth
- focus/blur
- viewport size
- coarse timing

These signals should be privacy-minimal and used to distinguish capability, not
to identify a person.

### 5. Crawl Graph

Generate deterministic crawl paths with depth, branch, and trap links.

Record:

- deepest path visited
- traversal order
- time between page requests
- whether query strings are preserved
- whether nofollow or robots hints are respected

### 6. Consent And Interruption Behavior

Cookie-consent and interstitial fixtures should test whether AI search and
assistant browsing clients read through visible popups, execute accept/reject
flows, ignore the UI, or report only the consent layer. Compare the result
against server logs, tracking-pixel hits, analytics events, and subresource
fetches.

### 7. Network Provenance

IP classification should be pluggable:

- reverse DNS
- ASN lookup
- cloud/datacenter CIDR list
- known crawler published ranges
- Tor/VPN/proxy lists where legally and ethically appropriate

Do not hard-code "datacenter = bot". Store the raw lookup result and a score.

## Classification Rule

Each visit should get:

- `claimedIdentity`: what the user-agent says
- `networkHints`: what IP/ASN/rDNS suggests
- `capabilityHints`: what resources and JS events show
- `behaviorHints`: traversal depth and timing
- `classification`: best current label
- `confidence`: 0.0 to 1.0
- `explanation`: short human-readable reason

## Measurement Surface Isolation

The lab has two kinds of surfaces:

- Measurement surfaces: `/lab/...`, `/track/...`, crawler fixtures, resource
  probes, and client capability endpoints.
- Observer surfaces: `/dashboard/...`, findings pages, and human-facing
  research review tools.

Measurement surfaces should stay plain and server-rendered by default. Do not
let dashboard styling, frontend framework assets, app shells, hydration scripts,
or visual-system resets leak into crawler-observed pages. Aesthetic or rendering
behavior belongs in an experiment page only when it is the explicit thing being
tested, such as JavaScript rendering, CSS-hidden text, image-only content, or
layout/media behavior.

Observer surfaces may use richer UI tooling as long as they consume existing
evidence APIs and do not change what crawlers see.

## Finding Requirements

A finding is valid only when it includes:

- hypothesis
- exact test URL
- exact prompt or triggering action
- timestamp window
- raw request evidence
- expected result
- observed result
- repeat instructions
- limitations and possible false positives

Document findings even when the result was not the result the test was meant to
produce. A failed setup, unreachable endpoint, IPv4/IPv6 mismatch, stale DNS
record, certificate issue, cache artifact, no-hit attempt, or model claim that
does not match origin evidence can be a valid lab finding. Label it as a
reachability, infrastructure, policy, or negative-evidence finding instead of
discarding it because the original content-behavior test did not complete.

If the observation is meaningful enough to mention in a work summary, it should
have a discoverable home in the lab: a numbered finding, an operational finding
in `FINDINGS.md`, a relevant domain doc, or a `Needs Verification` entry.

## Ethics And Safety

- Publish a research notice.
- Avoid collecting more personal data than needed.
- Hash or truncate IPs in public dashboards unless full IPs are explicitly
  required in a private research environment.
- Do not use deceptive downloads, exploit payloads, or invasive fingerprinting.
- Prefer capability fingerprints over personal fingerprints.