# Data schema of the collector

*Who May Read the Web?* (October 2026). This file describes what `collect.mjs` writes. It is the contract between collection and analysis: every field an analysis may rely on is listed here, with the rule that produced it.

Schema identifiers: site record `wmrw-site-record/1`, run manifest `wmrw-run-manifest/1`, sampling frame `wmrw-sampling-frame/1`, codebook `wmrw-codebook/1`.

## 1. Files in a data directory

All paths are relative to `--data-dir`.

| Path | Written by | Content |
| --- | --- | --- |
| `tranco/tranco_647LX-1m.csv`, `tranco/tranco_647LX-1m.meta.json` | `sample.mjs` | Cached Tranco list and its download record (time, URL, sha256) |
| `records/part-NNNNNN.jsonl.gz`, `records/part-NNNNNN.jsonl` | `collect.mjs` | Site records, one JSON object per line, lines separated by LF (`\n`) only. A full shard is gzipped; the last shard of a run stays plain |
| `run-manifest.json` | `collect.mjs` | One entry per invocation: times, Node version, code hashes, configuration, counts |
| `journal.jsonl` | `collect.mjs` | One `{"t","event":"start","domain"}` line per site, written before the site is contacted |

Read records with `readRecords(dataDir)` from `collect.mjs` (an async generator that handles both shard forms). Any other reader must split at LF only: a record may hold U+2028 or U+2029 inside a JSON string (site content; `JSON.stringify` leaves them unescaped), and line readers that also break at those code points, such as Node's `readline` or Python's `str.splitlines()`, tear such a record apart. **A domain appears in at most one record.** A sampled domain without a record was not reached: either the run has not got to it, or a crashed session had started it (those domains are listed in `sessions[].interruptedDomains` of the manifest). Denominators must therefore come from `data/sample.csv`, never from the number of records.

## 2. Site record

```jsonc
{
  "schema": "wmrw-site-record/1",
  "status": "collected",            // see 2.1
  "domain": "example.com",          // as in sample.csv
  "rank": 223, "stratum": "S1", "sampleOrder": 1,
  "collector": "1.0.0",
  "startedAt": "2026-10-..T..Z", "finishedAt": "2026-10-..T..Z",
  "origin": { "chosen": "https://example.com", "candidates": [ ... ] },   // 2.2
  "plan": [ "robots-txt-browser", "robots-txt", ... , "rsl-license" ],   // planned order of the 26 probes
  "robotsOrder": [ "robots-txt-browser", "robots-txt" ],                 // first two entries of plan
  "stop": null,                     // 2.3
  "refusalCheck": null,             // 2.3
  "ownPolicy": { ... },             // 2.4
  "robotsPair": { ... },            // 2.5
  "controls": { "softRoot": false, "softWellKnown": false },             // 2.6
  "counts": { "urlsRequested": 25, "wireRequests": 27, "rateLimitEpisodes": 0 },  // 2.7
  "requests": [ /* exactly 26 entries, in the order of config.mjs PROBES (n = 1..26) */ ],
  "complete": true
}
```

### 2.1 `status`

| Value | Meaning | Other fields |
| --- | --- | --- |
| `collected` | The site was processed. This says nothing about success: a dead domain is `collected` with failed requests | all of the above |
| `excluded` | The domain is on the opt-out list; it was never contacted. **Must not be published per domain** | identity fields only |
| `invalid-domain` | The list entry is not a usable host name | identity fields only |
| `collector-error` | A defect in the collector stopped this site; `error` holds the message. The site is not requested again | identity fields, `error` |

### 2.2 `origin`

`chosen` is the origin every probe URL is built from, or `null` when no origin could be connected to. `candidates` lists the attempts in order, each `{ origin, reason, by, outcome, code }`:

- `reason`: `https-apex`, `https-www-fallback`, `http-apex-fallback`, `http-www-fallback`;
- `by`: which of the two robots.txt probes made the attempt;
- `outcome`: `connected`, or the error class of the connection failure (`dns-error`, `connect-error`, `tls-error`, `timeout`, `blocked-address`); `code` is the system error code.

The next candidate is tried only when no connection could be made (DNS, TCP or TLS failed), so at most one origin receives an HTTP request during discovery. When the first identity finds no origin, the second identity runs the same discovery again, which is why up to eight candidates can be listed.

### 2.3 `stop` and `refusalCheck`

`stop` is `null` when every allowed probe was attempted. Otherwise it names why the remaining probes were not sent; those probes carry the same value as `notFetchedReason`.

| `stop` | Trigger |
| --- | --- |
| `origin-unreachable` | No origin candidate could be connected to |
| `robots-unreachable` | The research identity's robots.txt request ended in a 5xx or a network failure. RFC 9309 section 2.3.1.4 says to assume complete disallow |
| `edge-refused` | The research identity's robots.txt was refused (see below) and the home page was not served either |
| `rate-limited-stop` | A request was answered 429 again after the one permitted retry, or the server asked for a wait above 30 s, or a second 429 episode occurred |

`refusalCheck` is `{ robots, home }` (two reachability codes) when the research identity's robots.txt came back as `edge-block`, `edge-challenge` or `forbidden-origin`. In that case the home page is requested next, out of the planned order, and the site is left alone unless that request is `ok`. It is `null` otherwise.

Whatever stops a site, both robots.txt requests of the pair are always made.

### 2.4 `ownPolicy` — how the crawler read the site's robots.txt for itself

| Field | Meaning |
| --- | --- |
| `basis` | `rules` (2xx: the body was parsed and obeyed), `allow-all` (4xx or exhausted redirects), `disallow-all` (5xx, network failure, no origin), `none` (the site was stopped by a 429 before any decision) |
| `rfc9309Reading`, `googleReading`, `readingBasis` | The two readings of the fetch outcome (section 5.3) |
| `addressedToUs` | Some group's `User-agent` value contains `WhoMayReadTheWeb-Research` |
| `starPresent` | The applicable rules came from the `*` group |
| `disallowedProbes` | `[{ id, rfc, lenient }]` for every probe that was not requested because of a rule. `rfc` and `lenient` are `{ allowed, source, rule }` under the RFC 9309 reading and under Google's lenient reading |

The crawler stays out when **either** reading disallows a path, and it treats a group as addressed to it when the `User-agent` value merely contains its token. This is deliberately stricter than RFC 9309: an operator who tried to opt out should not be ignored over a typo. Count "sites that disallowed us under RFC 9309" with `rfc.allowed === false`.

### 2.5 `robotsPair` — the paired reachability comparison

`{ research, browser, researchStatus, browserStatus, sameBody, researchValid, browserValid }`. `research` and `browser` are reachability codes (section 4); `sameBody` is `true`/`false` when both requests returned a body and `null` otherwise; `*Valid` is the strict-tier verdict of each body.

The two requests go to the same URL and the same IP address (the address of a host is fixed for the run, and replaced for all later requests only if it cannot be connected to), and carry identical headers in identical order except `User-Agent`. Their order is `robotsOrder`, fixed by one bit of `sha256(orderSalt:robots-order:domain)`.

### 2.6 `controls`

`softRoot` / `softWellKnown`: `true` when the random control path at that level answered 200, `false` for any other status, `null` when the control was not requested or got no HTTP answer.

### 2.7 `counts`

`urlsRequested` is the number of probes that were attempted (at most 26; an attempt that failed in DNS counts). `wireRequests` is the number of GETs for which a connection was attempted, including redirect hops, the single retry, origin-discovery attempts and attempts on a further address of the same host. `rateLimitEpisodes` is the number of 429 answers received.

## 3. Request entries

`requests[i]` describes probe `n = i + 1`. Probe identifiers, paths and headers are in `config.mjs` (`PROBES`) and `codebook.json` (`probes`).

### 3.1 A probe that was not requested

```jsonc
{ "id": "llms-txt", "n": 7, "seq": null, "url": "https://example.com/llms.txt", "method": "GET", "ua": "research",
  "fetched": false, "notFetchedReason": "disallowed", "reachability": "not-fetched" }
```

`notFetchedReason`: `disallowed`, `robots-unreachable`, `edge-refused`, `rate-limited-stop`, `origin-unreachable`, or `no-license-pointer` (only for `rsl-license`: robots.txt has no `License:` line with an absolute URL on a host of the sampled domain or on the host that served robots.txt). `url` is `null` when it was never determined.

### 3.2 A probe that was requested

| Field | Meaning |
| --- | --- |
| `id`, `n` | Probe identifier and its number on the study page |
| `seq` | Position among the requests actually made for this site (1 = first) |
| `url`, `method`, `ua` | Requested URL, always `GET`, `research` or `browser` |
| `startedAt`, `elapsedMs` | Start of the logical request and its total duration, including rate-limit waits, redirect hops and a retry |
| `finalUrl` | URL of the last hop |
| `status` | Final HTTP status, or `null` when no HTTP response was received |
| `redirects` | Redirects followed (0–5) |
| `redirectStop` | `null`, or why the chain ended on a 3xx: `limit` (a sixth redirect), `loop`, `cross-site`, `disallowed` (the target path is disallowed for us), `bad-location` (no usable `Location`: missing, not http(s), or with credentials). `notFollowed` holds the refused target for `cross-site` and `disallowed` |
| `retries` | Extra GETs after a network failure or a 429 (0 or 1) |
| `rateLimit` | Present after a 429: `{ retryAfterMs, waitedMs, retried, stillLimitedAfterRetry }` |
| `hops` | Every GET in order: `{ url, status, location, remoteIp, elapsedMs, error, retried?, addressFallbacks? }`. `elapsedMs` here is wire time only. `addressFallbacks` is present when the TCP connection to the host's first address failed and a further address of the same host was used (at most two further addresses; the host then keeps that address for the rest of the run) |
| `error` | `null`, or the failure of the last GET: `{ class, code, phase, message }` with `class` one of `timeout`, `tls-error`, `dns-error`, `connect-error`, `protocol-error`, `blocked-address` and `phase` one of `dns`, `connect`, `tls`, `headers`, `body`. A failure in the `body` phase also carries `statusSeen`, the status line that had arrived; the request still counts as failed, because an incomplete robots.txt must not be read as a complete one |
| `remoteIp` | Address that answered (or was tried last) |

When an HTTP response was received, the entry also has:

| Field | Meaning |
| --- | --- |
| `httpVersion`, `tlsProtocol`, `ttfbMs` | Protocol facts of the final hop |
| `headers` | Response headers on the allow-list `RECORDED_RESPONSE_HEADERS`, lower-case names; a repeated header is an array. Long values are cut and end in `[cut]` |
| `setCookieCount` | Number of `Set-Cookie` headers. Cookie names and values are never stored |
| `servedBy` | `{ vendors, botManagers }`: CDN / hosting hints from header names, and bot-management hints derived from cookie names before they are dropped. Hints, not proof (see `codebook.json`, `fingerprints.edgeHeaderHints`) |
| `edge` | `null`, or the edge page that was recognised: `{ kind: "edge-challenge" \| "edge-block", vendor, signature, evidence }` |
| `redirect` | Requested URL against final URL: `{ crossHost, schemeChanged, pathChanged, elsewhere, sameBaseName, toRoot }`. `elsewhere` is the E.3 rule (the final path differs) |
| `body` | Section 3.3 |
| `validation` | Section 3.4 |
| `soft404` | For robots.txt, file and licence probes: `{ checked, match, against }`. `match` is `hash` (same bytes as a reference), `normalized` (same after removing the echoed path, long digit-bearing tokens and digit runs) or `null`; `against` is `control-root`, `control-well-known` or `home-html`. `checked` is `false` when no reference body exists |
| `reachability` | Section 4 |
| `readings` | robots.txt probes only: section 5.3 |
| `robots` | robots.txt probes with a 2xx status: section 5. `{ "sameAs": "robots-txt" }` on the browser probe when its body is byte-identical to the research probe's |
| `headerSignals`, `homeSignals` | The two home-page probes only: section 6 |

### 3.3 `body`

| Field | Meaning |
| --- | --- |
| `bytes` | Length of the decoded body that was read (at most 1,000,000) |
| `truncated` | The body was cut at the cap |
| `wireBytes` | Bytes received before decoding |
| `sha256`, `normalizedSha256` | Hash of the bytes, and of the soft-404 normal form |
| `isHtml`, `utf8` | E.3 HTML test; whether the bytes are valid UTF-8 |
| `saved` | `full`: the whole body is in `text` (valid UTF-8) or `base64`. `head`: only the first 2,048 bytes, in `base64`. `none`: nothing but the hashes (home pages, empty bodies). `same-as-robots-txt`: identical to the research identity's robots.txt, which holds the text |
| `decodeError`, `unsupportedEncoding` | Present when `Content-Encoding` could not be decoded |

A body is saved in full only when the status is 2xx and the body is not HTML. Use `storedBodyBytes(body)` from `collect.mjs` to get the bytes back.

### 3.4 `validation`

`{ tier, present, valid, strictPlus, rejections, mislabelled, structural, checks }`

- `tier`: `two-tier`, `lenient-only` (no strict rule could be verified: `ai-txt`, `agents-json`, `ai-plugin`; `valid` is always `null`), or `descriptive` (home page and controls).
- `present` (lenient tier) and `valid` (strict tier): the rules are stated per validator in `codebook.json` (`probes[].validatorRules`) and implemented in `validators.mjs`. `valid` implies `present`. Headline adoption figures should use `valid`.
- `strictPlus`: an optional third tier, `null` when the validator has none. Its meaning is in `checks.strictPlusMeans` (`valid-with-links`, `contact-and-expires`, `resource-origin-equals-probed-origin`, `issuer-origin-equals-probed-origin`).
- `rejections`: the generic rules that fired: `not-200`, `edge-challenge`, `edge-block`, `rate-limited`, `empty`, `html-body`, `redirected-elsewhere`, `truncated`, `soft-404`. An edge page, a redirect elsewhere, or a soft-404 match on a 2xx turns `present`, `valid` and `strictPlus` off.
- `structural`: `{ present, valid }` as the file's own validator saw the body, before the redirect and soft-404 rules. It lets an analysis measure what those two rules remove.
- `mislabelled`: the response said `text/html` but the body passed.
- `checks`: validator-specific facts (for example `fileClass`, `markdownLinks`, `issuerOrigin`, `topLevelKeys`).

`Content-Type` never decides presence, except for `home-markdown`, where the media type is the phenomenon being measured.

## 4. Reachability codes

One code per entry, in `reachability`. Precedence is top to bottom.

| Code | Rule | R2 notes E.5 name |
| --- | --- | --- |
| `not-fetched` | No request was sent | — |
| `timeout`, `tls-error`, `dns-error`, `connect-error`, `protocol-error`, `blocked-address` | The last GET failed; the code is `error.class` | timeout, tls-error, dns-error |
| `redirect-loop` | `redirectStop` is `loop` or `limit` | redirect-loop |
| `redirect-not-followed` | `redirectStop` is `cross-site`, `disallowed` or `bad-location` | — |
| `edge-challenge` | A challenge page was recognised, at any status | edge-challenge |
| `rate-limited` | Status 429 | rate-limited-429 |
| `edge-block` | A block page of an edge service was recognised | edge-block |
| `forbidden-origin` | Status 401 or 403 without a recognised edge signature. It can include edge blocks this instrument does not recognise; `servedBy` and the saved body head allow a hand check | forbidden-401/403-origin |
| `absent` | Status 404 or 410 | absent-404/410 |
| `client-error-other` | Any other 4xx | — |
| `server-error` | 5xx | server-5xx |
| `soft-404` | 2xx, and the body matches a control probe or the home page | soft-404 |
| `ok` | 2xx otherwise | ok |
| `unexpected-status` | Anything else (1xx, a 3xx that is not a redirect) | — |

`ok` describes the HTTP exchange, not the file: a 200 HTML page that differs from the controls is `ok` with `validation.present === false`. "The policy file was obtained" is `reachability === "ok" && validation.valid === true`.

At freezing, only three edge signatures were verified against a source (`cf-mitigated: challenge`, and the Cloudflare titles `Just a moment...` and `Attention Required! | Cloudflare`). After the collection, seven more were raised to `vendor-documented` under the protocol's section 10 rule (a cited source and a hand check of the matching responses); those entries carry `sourceUrl`, `verifiedOn` and `handCheck` in `codebook.json`. The remaining six keep `evidence: "implementer-recall"` and must be checked against saved samples before a count based on them is reported; one of them, `cloudfront-request-blocked`, was raised and withdrawn the same day (the two reviewers agreed on 16 of its 20 drawn files, below the 19 required) and keeps its `sourceUrl` and `handCheck` for the record, without `verifiedOn`. The `edge` field of a record holds the evidence level at collection time, not the current one; `derive.mjs` reads the current level from the codebook.

## 5. robots.txt analysis (`requests[].robots`)

Produced by `analyzeRobots()` in `robots.mjs` from the body, under the RFC 9309 profile, for the 54 codebook tokens plus `WhoMayReadTheWeb-Research`.

| Field | Meaning |
| --- | --- |
| `totalBytes`, `parsedBytes`, `flags` | `flags`: `bom`, `truncated` (longer than 512,000 bytes; only complete lines within the limit were parsed), `nonUtf8`, `nulBytes`, `crLineEndings`, `crlfLineEndings` |
| `fileClass` | `has-directives`, `no-directives`, `cf-policy-comment-only`, `empty`. "Has a robots.txt with rules" must mean `has-directives` |
| `directiveLines`, `lines`, `syntax`, `groups` | Line counts by kind; syntax anomalies (`emptyDisallow`, `patternWithoutLeadingSlash`, `userAgentWithComma`, `userAgentNotRfcToken`, `groupingSensitive`); number of groups and of distinct `User-agent` values |
| `star` | The merged `*` groups: `{ present: false }` or `{ present: true, groups, managed, level, rootAllowed, fullDisallowStrict, allowRules, disallowRules }` |
| `named` | The same summary for every listed token that some group names, keyed by the token as spelled in the codebook. A token that is absent here is not named |
| `contentSignals`, `contentUsage`, `licenses`, `agentmaps`, `aiTrainingLegacy` | Non-REP records, each with `line`, `position` (`before-first-group` or `in-group`), `agents` and `star` of its group, and `inManagedBlock`. Content-Signal has `path`, `signals`, `unknownKeys`; Content-Usage has `path`, `preferences`, `unknown`. The two are never translated into each other. An absent key means the operator said nothing about that use |
| `sitemaps`, `crawlDelay`, `unknownFields` | Counts; `unknownFields` is a histogram of unrecognised field names (at most 40 names, then `(other)`) |
| `fingerprints`, `fingerprintDetails`, `managedBlocks` | Detected fingerprint ids (catalogue with evidence levels in `codebook.json`); `managedBlocks` are `{ id, vendor, intermediary, label, beginLine, endLine, closed }` |
| `withoutIntermediary` | `null`, or the verdicts recomputed with every intermediary-written block removed: `{ fileClass, groups, star, named, contentSignals, contentUsage, licenses }` |
| `sensitivity` | Differences from the primary reading; see 5.2 |

### 5.1 The two counting rules

Use the helpers exported by `robots.mjs`; they read an analysis object and need no body.

- `explicitVerdict(analysis, token)` → `{ named, label }` with `label` one of `full-disallow`, `partial-disallow`, `explicit-allow`, `not-named`. Only groups that name the token count.
- `effectiveVerdict(analysis, token)` → `{ source, label, level, rootAllowed, ... }` with `source` one of `named`, `star`, `none`; `label` one of `full-disallow`, `partial-disallow`, `explicit-allow`, `inherits-star`, `no-rules`; `level` one of `full-disallow`, `partial-disallow`, `no-restriction`. This is what a compliant crawler ends up with after the `*` fallback.

Definitions: `full-disallow` means the path `/` is disallowed. `fullDisallowStrict` is the stricter reading: `/` is disallowed by a pattern that matches every path and the group has no non-empty `Allow` rule. `partial-disallow` means `/` is allowed and at least one non-empty `Disallow` is not cancelled by an identical `Allow`. A group with no rules, an empty `Disallow`, or only `Allow` rules is `no-restriction` (`explicit-allow` when it names the token).

`managed` says where the groups behind a verdict sit: `all`, `some` or `none` of them inside an intermediary block. For "without intermediary injection" figures use `withoutIntermediary` when it is not `null` and the primary fields otherwise. The two can differ in the permissive direction too: Cloudflare's managed `*` group carries `Allow: /`, which ties with, and therefore overrides, an origin `Disallow: /` in the merged group.

### 5.2 `sensitivity`

Each part lists only what differs from the primary reading (`{}` and `null` mean no difference).

| Key | What changes |
| --- | --- |
| `googleAgentMatch.named` | Tokens are compared with the leading `[A-Za-z_-]` run of each `User-agent` value (Google's truncation) instead of the whole value. A value of `null` means the token is named in the primary reading but not here |
| `commaAgentMatch.named` | Comma-separated `User-agent` values are split (RFC 9309 erratum 8895, status Reported) |
| `googleProfile` | The whole file read as Google's open-source parser reads it: `named`, `star`, and how many lines were accepted only through a typo or a missing colon |
| `extensionsCloseGroups` | `Crawl-delay`, `Content-Signal`, `Content-Usage` and the legacy AI-training records end a run of `User-agent` lines (the reading RFC 9309 rejects; Liu et al., IMC '25, App. B.2 case 3) |

To evaluate any other token or path, parse the saved body again: `parseRobots(storedBodyBytes(body))`, then `evaluate`, `tokenVerdict` or `analyzeRobots` with another token list.

### 5.3 `readings` of a robots.txt fetch

| Outcome | `rfc9309` | `google` |
| --- | --- | --- |
| 2xx | `follow-rules` | `follow-rules` |
| 4xx except 429 (401 and 403 included) | `allow-all` | `allow-all` |
| 429 | `allow-all` | `not-treated-as-missing` |
| 5xx, timeout, DNS, TLS or connection failure | `disallow-all` | `staged-server-error` |
| more than five redirects, redirect loop | `allow-all` | `allow-all` |
| any other status | `unspecified` | `allow-all` |

`basis` names the row. `staged-server-error`: Google stops crawling for 12 hours, then uses a cached copy for up to 30 days and, without one, assumes no restrictions.

## 6. Home page signals

`headerSignals` (both home probes): `{ contentSignal[], contentUsage[], xRobotsTag[], robots, links[], linkFlags, tdm, markdownTokens, wwwAuthenticate }`. `xRobotsTag` entries are `{ raw, bot, tokens }`; `robots` aggregates the entries that are not scoped to one crawler.

`homeSignals` (when the body is HTML, else `null`): `{ title, htmlLang, generator[], metaRobots[], robots, tdm, links[], linkFlags, rslInlineScripts[], jsonLd, dataNosnippetElements, counts, truncatedLists }`.

- `robots` (in both objects): `{ noai, noimageai, noarchive, nocache, nosnippet, noindex, nofollow, maxSnippet, disallowAiTraining, allowAiTraining }`. In `homeSignals` it aggregates `<meta name="robots">` only; crawler-specific meta elements stay in `metaRobots`.
- `links`: `{ tag, rel, href, type }` for the relations `license`, `api-catalog`, `describedby`, `ard`, `ai-catalog`, `service-desc`, `service-doc`, `service-meta`, `status`, and `alternate` only when its type is `text/markdown`. `linkFlags`: `{ rslLicense, anyLicense, apiCatalog, describedByLlmsTxt, markdownAlternate, ard, aiCatalog, serviceDesc, serviceDoc }`.

These are page-level signals of one page. They say nothing about the rest of a site.

## 7. Run manifest

`run-manifest.json` is `{ schema, study, sessions: [...] }`. Each session has `startedAt`, `endedAt`, `outcome` (`completed`, `stopped-early`, or `running` if the process died), `node`, `platform`, `argv`, `vantageNote`, `sample` (`path`, `sha256`, `selected`), `options`, `codeSha256` (one hash per code file and for `codebook.json`), `config` (every export of `config.mjs`), `counts` (`alreadyDone`, `collected`, `excluded`, `interrupted`, `invalidDomain`, `failed`), `interruptedDomains`, `remaining`, and `eventLoopDelayMs` (`max`, `p99`, `mean`: how long the single-threaded collector was blocked; a large `max` means that timings and timeouts of sites in flight at that moment are inflated).

## 8. Counting from these records

The fields were chosen so that the usual denominators can be stated without ambiguity. With `r` a site record and `q(id)` its request entry:

| Quantity | Definition |
| --- | --- |
| sampled | rows of `data/sample.csv` (per stratum: `stratum`) |
| processed | `r.status === "collected"` |
| answered at all | some entry has `status !== null` |
| origin found | `r.origin.chosen !== null` |
| robots.txt obtained by the research identity | `q("robots-txt").reachability === "ok" && q("robots-txt").validation.valid === true` |
| robots.txt lenient presence | `q("robots-txt").validation.present === true` |
| file adopted (strict) | `q(id).validation.valid === true`; lenient: `present === true`; for the three lenient-only files only `present` exists |
| probe not attempted | `q(id).fetched === false`, by `notFetchedReason` |
| paired outcome | the pair (`r.robotsPair.research`, `r.robotsPair.browser`) |

A probe that was not attempted is unknown, not absent: sites that disallowed the crawler, refused it, or were unreachable have no observation for the affected files, and rates for those files should say which denominator they use.

## 9. What the data cannot show

- **One vantage point, one pass, one day per site.** State the vantage (`vantageNote`) with every result.
- **The browser identity is a string, not a browser.** Both requests of the pair come from the same HTTP/1.1 client with the same TLS fingerprint and no JavaScript. A site that blocks on those grounds blocks both; the pair isolates the effect of the `User-Agent` string alone.
- **Registrable domains only.** Policy files on other hosts of a site are not seen, and a redirect to another registrable domain is not followed for anything but robots.txt.
- **IPv4 first.** A host is contacted over IPv6 only when it has no A record.
- **Root-level probes.** Path-suffixed OAuth metadata and sub-path `llms.txt` files are not probed.