# Who May Read the Web? — Pre-registered protocol

**Study.** *Who May Read the Web? Measuring How Top Websites Admit, Refuse, and Invite AI Crawlers and Agents (October 2026)*

**Author.** Max Song, independent developer (<https://maxsong.me/>). No institutional affiliation; no institutional ethics review. The design, the instrument and the analysis code were prepared with substantial help from AI coding agents. The author answers for every decision recorded here.

**Version.** 1.0, dated 2026-10-05.

**Status.** Written and frozen before any request was sent to a sampled website. Up to this date the collector has run only against a fixture server on `127.0.0.1`, and the analysis only on synthetic data with known answers. Section 15 lists the points that still need the author's confirmation at the moment of freezing.

This document fixes, in advance, what will be measured, how each quantity is defined, which numbers the paper will lead with, and which comparisons count as tests. Anything the paper reports that is not described here will be labelled exploratory. Anything done differently will be entered in the deviations log (section 11) with its reason.

---

## 1. Questions

Websites now tell automated readers two opposite things. In robots.txt many of them refuse the crawlers of AI companies. In a growing set of new files and signals (llms.txt, Content-Signal, API catalogs, agent cards) some of them invite AI systems in. The study measures both on one ranked sample, at one moment, with open per-site data.

| | Question |
| --- | --- |
| RQ1 Refusal | How do popular websites treat AI crawlers in robots.txt, token by token? Do they treat the purposes that operators document differently: collecting training data, building an AI search index, and fetching a page because a user asked? |
| RQ2 Authorship | How much of that robots.txt text did the site write, and how much was written by an intermediary or copied from a template? |
| RQ3 Invitation | How many sites publish files or signals that invite AI systems and agents, when a file is counted only if it is what it claims to be? |
| RQ4 Consistency | How often does one site both invite and refuse the same purpose? |
| RQ5 Reachability | Can a crawler that says what it is read the policy file at all, compared with a client that sends an ordinary browser string? |

The study is descriptive. It registers no hypothesis for the descriptive quantities. Six comparisons are registered as tests (section 7.4); what earlier work leads one to expect is stated there, and every test is two-sided.

### 1.1 What is already known

Every single quantity in this study has been published by someone for 2025 or 2026. The study does not claim to be the first to measure any of them. The closest earlier work:

- Liu et al., *Somesite I Used To Crawl* (IMC '25, DOI 10.1145/3730567.3732913), measured explicit robots.txt rules against 24 AI user agents on Tranco sites to October 2024, and user-agent-conditioned blocking of pages on the top 10,000.
- Hoffmann et al., *From robots.txt to ai.txt* (SIGCOMM CCR 56(2), July 2026, DOI 10.1145/3831956.3831960), measured robots.txt, ai.txt and llms.txt on about four million domains in October 2025. Only its abstract and its code have been read; none of its numbers is used here.
- Bouchaud and Ramaciotti (arXiv:2510.09031, preprint) reported disallow rates by agent and rank bucket on the Chrome UX Report top million for August and September 2025.
- Cloudflare Radar publishes weekly counts of AI-agent rules in the robots.txt of its top 10,000 domains, and weekly adoption counts of agent-facing files on about 160,000 popular domains. Cloudflare is also the author of the Content Signals proposal and operates a service that writes robots.txt text for its customers. Its figures are both a comparator and an interested source.
- AI Visibility (1 October 2026) scanned a vendor-defined set of AI files and AI crawler rules on about 1,900 domains, named the state "invites and refuses at once" a contradiction, and documented one llms.txt that was served to some user agents and not to others. It measures a file set that it proposes itself.

What the searches made for this study did not find is a single study that puts a purpose-aware typology of robots.txt, standards-track invitation files, a site-level rate of contradiction, and an identity-paired test of the policy file itself on one ranked sample with open per-site data. The paper will say "not found in the searches described", never "first".

---

## 2. Frozen inputs

The files below are fixed. A test in the repository compares each hash in this table with the file on disk, so a change to any of them cannot pass unnoticed: it has to be entered in section 11.

| Input | File (under `public/papers/who-may-read-the-web/`) | sha256 |
| --- | --- | --- |
| Sampling frame record | `data/frame.json` | `11fde3ade9a614a4111dbf581cfe8cd0f8d47e1eda17f061a8a88a69ed89885e` |
| Sample of 10,000 domains | `data/sample.csv` | `57eb4438c0735a95fe9ba1551a6d383ce939fd681d0c0eec8c6c2eb5b3631321` |
| Category labels of the sample | `data/categories.csv` | `12317fc50b0f36ea1dc1709162f793e49dfd66731df0a44090c6c073a6ba2d2d` |
| Codebook (tokens, probes, validation rules, fingerprints) | `code/codebook.json` | `af4a50eb17261b04b58979344f611f141453af6a695af963770555139fabe31c` |
| Constants of the collector | `code/config.mjs` | `863c56a908090aefee4aedd11a128d7d36fcbe66c19a9399d6a1bcffcaf33236` |

Also fixed, and identified by the commit that first contains this file: the collector (`code/collect.mjs`, `code/robots.mjs`, `code/validators.mjs`), the analysis (`code/categorize.mjs`, `code/derive.mjs`, `code/analyze.mjs`, `code/figures.mjs`), and the public study page <https://maxsong.me/papers/who-may-read-the-web/crawler/>, which states the upper bound of what the collector may request. The run manifest written by the collector records the hash of every code file at the time of each run.

The sampling frame is Tranco list **647LX** (generated 2026-10-04; 1,000,000 rows; sha256 of the downloaded file `23165e5e9e9ec2b6b186fb092c285f1943d51d25fb48d27d7fe1e0cad0bca959`; permanent address <https://tranco-list.eu/list/647LX/1000000>). Tranco: Le Pochat et al., NDSS 2019, DOI 10.14722/ndss.2019.23386.

---

## 3. Design

**Type.** A cross-sectional observational measurement: one pass over a stratified random sample, from one vantage point, in October 2026. Nothing is manipulated except the `User-Agent` header of one extra request per site (section 5.9).

**Population.** The registrable domains of Tranco list 647LX. Tranco averages five popularity sources over thirty days. Three of them include infrastructure names as well as websites, so a substantial share of entries serve no web content. The study therefore reports every rate against more than one denominator (section 5.1) and never replaces a sampled domain that does not answer.

**Strata and sample.**

| Stratum | Ranks | Domains in the list | Sampled | Sampling fraction |
| --- | --- | ---: | ---: | ---: |
| S1 | 1–1,000 | 1,000 | 1,000 | 1 (census) |
| S2 | 1,001–10,000 | 9,000 | 3,000 | 1/3 |
| S3 | 10,001–100,000 | 90,000 | 3,000 | 1/30 |
| S4 | 100,001–1,000,000 | 900,000 | 3,000 | 1/300 |

The bounds match the rank buckets used by the Chrome UX Report, the Web Almanac and Bouchaud and Ramaciotti. Within a stratum the domains are ordered by `sha256(salt + ":" + domain)` with the public salt `who-may-read-the-web/647LX/2026-10-04`, and the first *n* are the sample. Anyone can recompute the sample from the list (`sample.mjs --check`). The position of a domain in that order is its `sampleOrder`.

**Why 10,000.** The size was set by what a single person can responsibly request: at most 26 URLs per site, one pass. With 3,000 domains in a stratum, a 95% interval for a share of 10% is about ±1.1 percentage points over all sampled domains, and about ±1.7 points over 1,200 domains (40% of a stratum, a plausible number of sites with a robots.txt that has rules). For a share of 1% the figures are ±0.4 and ±0.6 points. When nothing is found among 3,000 domains, the exact upper bound is 0.12%. The stratum-weighted estimate for the top million has a half-width of about 1.0 point at 10% over all domains and about 1.5 points when 40% of domains are in the denominator. Counts near zero in the lower strata are expected for several files and are results in themselves.

**Unit.** The registrable domain as listed. The collector tries `https://<domain>`, then `https://www.<domain>`, then the same two over `http://`, and moves to the next only when no connection could be made. All probes of a site go to the origin found this way. Policy files on other hosts of the same organisation are not seen.

**Requests.** At most 26 URLs per site, each with one `GET` (Appendix A). Every request carries `WhoMayReadTheWeb-Research/1.0 (+https://maxsong.me/contact/)`, with one disclosed exception: `/robots.txt` is requested a second time with the string of a common desktop browser. No request ever carries the name of another organisation's crawler.

**Conduct.** The collector reads robots.txt first and requests nothing that the file disallows for its own token or, failing a group for that token, for `*`. It takes the stricter of two readings of the file (RFC 9309, and Google's more lenient parser), so that an operator who tried to opt out is not ignored over a typo. It waits at least one second after a request to a host has ended before it sends that host another, follows at most five redirects, retries at most once, cuts bodies at 1,000,000 bytes, sends no cookie, runs no JavaScript and attempts no challenge. It leaves a site alone when robots.txt answers 5xx or cannot be reached (RFC 9309 section 2.3.1.4), when both robots.txt and the home page are refused, and when the site keeps answering 429. Both robots.txt requests of the pair are always made.

**Vantage point.** One machine, one network, IPv4 preferred. The run manifest and the paper state where it is.

**Pilot.** The first 50 domains of every stratum by `sampleOrder` (200 domains) are collected first. Section 10 says what may change after the pilot and what may not.

---

## 4. Instruments

**Collector** (`collect.mjs`). Writes one record per sampled domain with exactly 26 request entries. The record format is `code/SCHEMA.md`.

**robots.txt parser** (`robots.mjs`). Written against RFC 9309: groups for the same token are merged, the longest matching rule wins, `Allow` wins a tie, matching of tokens is case-insensitive on the whole `User-agent` value. Earlier studies found home-grown parsers to be a source of error, so the parser is checked against Google's open-source parser (`google/robotstxt`, commit `22b355ff`). On 649 hostile files (1,037,102 file-token-path cells) a second profile of the same parser that imitates Google's implementation agrees with it on every cell. The primary profile differs on 51,787 cells, each of which is explained by one or more of twelve named differences between RFC 9309 and Google's implementation, and none is unexplained. The comparison will be repeated on the collected files.

**Validators** (`validators.mjs`). A file is never counted because the server answered 200. Each probe has a lenient rule ("present") and, where a rule could be verified against the file's specification, a strict rule ("valid"); Appendix A lists them. Before either rule applies, a response is rejected when it is an HTML page, is empty, was reached through a redirect that changes the path, is a recognised challenge or block page, or matches the body that the same site returned for a random path that should not exist (two control requests per site: one at the root, one under `/.well-known/`).

**Codebook** (`codebook.json`). 54 crawler tokens with two purpose labels each, the 26 probes with their rules, 23 fingerprints of vendor-written robots.txt text and 16 signatures of edge pages, each with an evidence level.

**Categories** (`categorize.mjs`). Section 5.10.

**Analysis** (`derive.mjs`, `analyze.mjs`, `figures.mjs`). `derive.mjs` turns each record into one row of a per-site table. `analyze.mjs` only counts that table. The output is a pure function of the inputs: no clock, no randomness, no network.

---

## 5. Definitions

Every definition below is implemented in `derive.mjs` under the name given in `code font`. Where this text and the code disagree, that is a defect to be fixed and logged, not a second definition.

### 5.1 Record states and denominators

Every sampled domain is in exactly one record state (`RECORD_STATES`):

| State | Meaning |
| --- | --- |
| `collected` | The collector processed the site. This says nothing about success: a dead domain is collected, with failed requests |
| `excluded` | The operator asked to be left out. Never contacted (or, for a request received later, removed). Counted per stratum only; no per-site row is published |
| `invalid-domain` | The list entry is not a usable host name |
| `collector-error` | A defect in the collector stopped this site |
| `no-record` | The run never reached the domain |

No sampled domain is ever dropped from "all sampled domains". The paper uses these denominators, and names the one in use next to every percentage:

| Denominator | Members | Used for |
| --- | --- | --- |
| all sampled domains | every row of `sample.csv`, whatever happened | adoption of files; outcome of the robots.txt request; the second version of every refusal figure |
| collected domains | record state `collected` | the paired comparison |
| domains with robots.txt rules | the research identity obtained a robots.txt with at least one directive (5.2) | refusal figures, policy types, authorship |
| domains that served a robots.txt file | the above, plus files without a directive | the two classes of files without a directive in the authorship table; the share of any intermediary-written text |
| domains with a valid invitation | at least one valid inviting file or granting signal (5.7) | conflicts |
| domains where the request was answered over HTTP | per file | a second reading of adoption that leaves out sites the crawler was not allowed or not able to ask |

The sizes of the first three are printed as `frame.sampled.sample`, `count.collected.s1` and `count.robots.s1` (and so on for the other strata).

A probe that was not sent is unknown, not absent. Sites that disallowed the crawler, refused it or could not be reached have no observation for the affected files. They stay in "all sampled domains", so adoption over that denominator is a lower bound; the number of such sites is reported for every file (for example `invite.llms-txt.not-attempted.all.s1`).

### 5.2 Outcome of a robots.txt request

Each of the two robots.txt requests of a site falls into exactly one of seven groups (`ROBOTS_OUTCOMES`). The first three are "answered": the server gave a definitive answer about its policy.

| Group | Answered | Detail codes | Meaning |
| --- | --- | --- | --- |
| `rules` | yes | `rules` | status 200, not HTML, and at least one line with a robots.txt field (`User-agent`, `Allow`, `Disallow`, `Sitemap`, `Crawl-delay`, `Content-Signal`, `Content-Usage`, `License`). A file that only lists a sitemap is in this group |
| `no-rules` | yes | `cf-comment-only`, `comments-only`, `empty` | a file was served that has no such line: comments only, among them the policy-only text of an intermediary, or nothing |
| `absent` | yes | `http-404`, `soft-404`, `not-a-robots-file`, `redirected-elsewhere` | the server answered and there is no robots.txt: 404 or 410; a 200 whose body equals the site's answer for a random path or its home page; an HTML page; a redirect to another path |
| `refused` | no | `forbidden-origin`, `edge-block`, `edge-challenge`, `rate-limited` | 401 or 403, a recognised block or challenge page, or 429 |
| `error` | no | `server-error`, `client-error-other`, `unexpected-status`, `redirect-loop`, `redirect-not-followed` | an HTTP answer that settles nothing |
| `unreachable` | no | `timeout`, `tls-error`, `dns-error`, `connect-error`, `protocol-error`, `blocked-address`, `not-fetched` | no HTTP answer |
| `not-collected` | no | `excluded`, `invalid-domain`, `collector-error`, `no-record` | the domain has no collected record |

Three points need saying.

1. RFC 9309 reads a 4xx answer to the robots.txt request, 401 and 403 included, as "no restrictions", and a 5xx or a network failure as "disallow everything". The study does not merge these into "no robots.txt". It reports the outcome as observed, and separately how many refused requests the RFC would read as permission (`robots.research.refused-reads-allow-all.sample`).
2. A robots.txt reached through a redirect that changes the path is classed `absent / redirected-elsewhere`, by the generic rule that guards every probe against catch-all redirects. RFC 9309 section 2.3.1.2 would let a crawler use such a file. The number of sites where the redirected body is, structurally, a robots.txt is reported (`robots.research.redirected-file.sample`); see section 15.
3. Only three signatures of edge pages were verified against a source (all Cloudflare). `forbidden-origin` can therefore contain block pages of other vendors. The split between `forbidden-origin` and the edge codes is reported only for verified signatures (`robots.research.refused-verified-edge.sample`); the group `refused` does not depend on it.

"Sites with robots.txt rules" always means: the request made under the **research identity** ended in the group `rules`. The file obtained under the browser identity is used only in the paired comparison and in one sensitivity analysis.

### 5.3 Reading a robots.txt

For a crawler token and a file, three levels:

- **full disallow**: the path `/` is disallowed for the token;
- **partial disallow**: `/` is allowed, and at least one non-empty `Disallow` rule is not cancelled by an identical `Allow`;
- **no restriction**: anything else (no rule, an empty `Disallow`, only `Allow` rules).

and two counting rules:

- **explicit**: only groups that name the token count. A token that no group names has no explicit level. This is the rule of Liu et al. and of Cloudflare Radar's "fully disallowed".
- **effective**: what a compliant crawler ends up with. Its own group if one names it; otherwise the `*` group; otherwise no rule (RFC 9309 section 2.2.1). This is the rule of AI Visibility and of Fletcher's news factsheet.

Every refusal figure is produced under both rules and over two denominators (domains with robots.txt rules; all sampled domains). The four cells let the numbers stand beside studies that made different choices without mixing definitions.

A stricter reading of "full disallow" (every path disallowed and no `Allow` rule in the group) and the reading of Google's parser are sensitivity analyses (section 9).

### 5.4 Tokens and purposes

The codebook lists 54 tokens. Two of them, `Googlebot` and `bingbot`, are general web search crawlers whose operators document no separate token for AI search: a site cannot refuse the AI use without leaving the search engine. They are reported as a baseline and belong to no AI class (`BASELINE_SEARCH_TOKENS`). The other 52 are the **AI tokens**.

Each token has two purpose labels.

- **First-party scheme (primary).** The purpose is what the operator's own documentation says. Classes: `training` (7 tokens), `search-index` (11), `user-fetch` (10). The remaining 24 AI tokens have a mixed or undeclared purpose and form the class `other`; CCBot, Google-Extended, Bytespider and meta-externalagent are among them.
- **Community scheme (secondary).** Derived mechanically from Cloudflare's published bot categories and the list in its managed robots.txt block: `training` 18, `search-index` 4, `user-fetch` 6. The other 24 AI tokens have no community label and form its class `other`. The two schemes put 27 tokens in different classes: 12 that the first-party scheme leaves in `other` and the community lists call training or search (CCBot, Google-Extended and Bytespider among them); 12 that operators document and the community lists do not cover; and 3 that both label, differently (DuckAssistBot, PetalBot, Diffbot).

Appendix B lists the sets. The paper's main text uses the first-party scheme; the community scheme appears beside it wherever a purpose is involved.

One blind spot is structural. Google's control over the use of content in its AI search features is a setting in its Search Console, not a crawler token. Nothing in this study can see it. Statements about Google are limited to `Google-Extended`.

### 5.5 Site-level stance and policy types

For a set of tokens and a counting rule, a site's **stance** is `none`, `some` or `all`, according to how many tokens of the set it fully disallows (`stance`).

The **policy type** of a site with robots.txt rules follows from three facts, namely whether at least one token of `training`, of `search-index` and of `user-fetch` is fully disallowed, and from two tie-breakers (`policyType`):

| Training | AI search | User-triggered | Type |
| :---: | :---: | :---: | --- |
| no | no | no | `open` if no token of `other` is refused either, else `other-only` |
| yes | no | no | `training-only` |
| yes | no | yes | `training-user` |
| yes | yes | no | `training-search` |
| yes | yes | yes | `blanket` if every token of all three classes is refused, else `all-purposes` |
| no | yes or no | yes or no, not both no | `not-training` |

The eight types are exhaustive and mutually exclusive among sites with rules. Two more cover the rest of the sample: `no-robots-rules` (the request was answered and no rule is stated) and `unknown` (refused, failed or not collected). The primary version uses the first-party scheme and the effective rule; the explicit rule and the community scheme are also produced.

### 5.6 Who wrote the robots.txt

Cloudflare writes robots.txt text on behalf of customers: a marked block of rules that it inserts into the origin's file, and a policy-only text that it serves for sites that have no file. A refusal that such a block contains is a product default as much as a decision of the site. So:

- An **intermediary block** is the text between a `# BEGIN Cloudflare <label>` line and the matching `# END Cloudflare <label>` line.
- Every refusal figure, every policy type and every signal figure is produced twice: for the file **as served**, and for the file with intermediary blocks removed (**own text**). The two can differ in both directions. The managed block's `*` group carries `Allow: /`, which merges with the origin's `*` group and can override an origin `Disallow: /`.

Each served robots.txt body gets one **authorship class**, the first that applies (`AUTHORSHIP_CLASSES`):

| Order | Class | Rule |
| ---: | --- | --- |
| 1 | `intermediary-comment-only` | the policy-only text of an intermediary; no directive |
| 2 | `intermediary-only` | every directive sits inside an intermediary block |
| 3 | `intermediary-plus-own` | an intermediary block, and directives outside it |
| 4 | `platform-template` | contains the default text of a content-management system or site builder, by a verified fingerprint |
| 5 | `generator-template` | contains a published block list or the output of a generator, by a verified fingerprint |
| 6 | `unverified-template` | matches only fingerprints that have not been verified |
| 7 | `no-known-template` | directives, and none of the known templates: written by the site, or by a tool this study does not recognise |
| 8 | `no-directives` | comments or nothing, not from a known intermediary |

A fingerprint is **verified** when its evidence level in the codebook is neither `implementer-recall` nor `needs-capture`. At the time of freezing, seven of the 23 fingerprints were checked against a source during the research phase and one is described in vendor documentation; the twelve default texts of content-management systems and three others are not verified. Until a fingerprint is verified (section 10), a file that matches it is counted in `unverified-template` and in no named class. Class 7 is a residual: it does not show that a person at the site wrote the file.

Among the sites that explicitly refuse at least one AI token, the study also reports how many do so only inside an intermediary block (`authorship.ai-refusal.intermediary-only.s1`).

### 5.7 The invitation layer

Twenty-one files and behaviours are probed (`ARTIFACTS`; Appendix A). They are reported in three groups and never added across groups without saying so:

- **registered**: a published specification with a permanent registration of the path: the API catalog of RFC 9727 and the A2A Agent Card;
- **proposal**: proposals, drafts, vendor features and industry specifications still in flux: `llms.txt`, `llms-full.txt`, Markdown content negotiation, the legacy Agent Card path, three candidate paths for MCP discovery, Agentic Resource Discovery, the Agent Skills index, `auth.md`, `agents.json`, `ai-plugin.json`, `ai.txt`, the TDMRep file and the RSL licence document;
- **infrastructure**: OAuth and OpenID metadata and `security.txt`. These are relevant to agents and are not an invitation to them. `security.txt` is a baseline with a known adoption history.

Each file is counted three ways, over all sampled domains:

- **status 200**: the request ended with status 200, whatever the body. This is what a scan of status lines would report;
- **present** (lenient tier): passes the generic rejections and the file's lenient rule;
- **valid** (strict tier): also passes the file's strict rule. **This is the figure the paper reports.**

The ratio of the first count to the third is reported per file (for example `invite.llms-txt.overcount`). For three files (`agents.json`, `ai-plugin.json`, `ai.txt`) no strict rule could be verified against a specification; they have a lenient count only and are never part of "valid". The Markdown request goes to the home page, which answers 200 at every live site, so no status-200 count is given for it.

**Signals inside robots.txt.** A `Content-Signal` record (Cloudflare's Content Signals Policy), a `Content-Usage` record (the IETF AIPREF drafts), a `License` record (RSL 1.0), an `Agentmap` record, and the records of an expired draft (`DisallowAITraining`, `AllowAITraining`) are counted as present when the file carries at least one. The two vocabularies are never translated into each other. The **site-wide value** of a key is the value stated by a record that sits in the `*` group or before the first group and carries no path other than `/` (`summarizeSignals`). When two such records disagree the value is `conflict`. A key that no site-wide record states is unstated, which is not "no".

**Which artifacts invite whom.** For the consistency analysis each inviting artifact addresses one audience:

| Audience | Tokens that stand for it | Invitations |
| --- | --- | --- |
| training | class `training` | `Content-Signal: ai-train=yes`; `Content-Usage: train-ai=y` |
| inference (reading at answer time) | classes `search-index` and `user-fetch` | `llms.txt`, `llms-full.txt`, Markdown content negotiation; `Content-Signal: ai-input=yes`; `Content-Usage: ai-use=y` |
| agent (acting for a user) | class `user-fetch` | API catalog, Agent Card (both paths), MCP server card, `mcp.json`, AI Catalog, Agentic Resource Discovery, Agent Skills index, `auth.md` |
| classic search | `Googlebot`, `bingbot` | `Content-Signal: search=yes`; `Content-Usage: search=y` |

Rights reservations (TDMRep), licence terms (RSL), the three lenient-only files and the infrastructure group invite no one and take no part in the conflict analysis. A site **has a valid invitation** when it has at least one valid inviting file or at least one site-wide signal that grants training or inference (`invite.any.strict.all.s1`). The classic-search signals are reported on their own and are not part of that set.

### 5.8 Invitation against refusal

Three quantities, from narrow to broad. Only a site with robots.txt rules can refuse anything, so only such a site can be in conflict.

- **Same-purpose conflict (primary).** The site has a valid invitation, and its robots.txt refuses at least one listed crawler of the audience that this invitation addresses. For a file the test is exact: is *the file's own path* disallowed for that crawler? For a signal the test is the full disallow of `/` under the effective rule (`invitationConflicts`).
- **Void invitation.** As above, and *every* listed crawler of that audience is refused.
- **Co-occurrence (broad).** The site has a valid invitation and fully disallows at least one AI token under the effective rule, whatever the purpose. A site that publishes an `llms.txt` for answer-time use and refuses training crawlers is counted here and is not a same-purpose conflict: the two statements can be consistent.

All three are reported as a count and as a share of the sites with a valid invitation, under both purpose schemes. The path-level test has a consequence that must be stated with the result. This study's crawler obeys robots.txt. A site whose `*` group disallows everything, or disallows the path of an invitation file, was never asked for that file (unless a group of its own lets this study's crawler in), so a conflict of that kind cannot be observed. The measured rate covers conflicts written against named crawlers by sites that left the path open to an unnamed one.

### 5.9 Reachability and the paired comparison

Each site receives two requests for `/robots.txt` at the same URL and the same IP address, with identical headers in identical order except `User-Agent`. One carries the declared research identity, the other the string of a common desktop browser. Which goes first is fixed per domain by one bit of a hash, so that order is balanced and cannot be chosen.

The outcome compared is **answered** (5.2). A site is concordant when both requests were answered or neither was, and discordant otherwise: answered for the browser identity only, or for the research identity only. The comparison is made among collected domains.

What this isolates is the effect of the `User-Agent` string alone. Both requests come from the same HTTP/1.1 client, with the same TLS fingerprint and without JavaScript. A site that blocks on those grounds blocks both, and the pair cannot see it. The browser identity is a header, not a browser.

Two checks accompany the comparison: the share of refused requests by which identity asked first (`pair.order.research-first.research-refused.sample`, `pair.order.browser-first.research-refused.sample`), and, among sites where both identities received a body, how often the bytes differ (`pair.different-body.sample`) and how often the two files refuse different AI tokens (`pair.different-rules.sample`).

### 5.10 Categories and the EU proxy

Categories come from open sources only, so that anyone can rebuild them. No proprietary classification is used.

- **Wikidata.** An item with an official website (property P856) that is an instance of a chosen class or of one of its subclasses, and is not dissolved. A website labels a sampled domain only when it *is* that domain: its host is the domain or `www.` plus the domain, and its path is `/`. A website on a path or a subdomain of the sampled domain is recorded as weak evidence and labels nothing. (Without this rule a university's page on a hosting platform would label the platform.)
- **Domain-name rules.** Namespaces reserved for government or education (`.gov`, `.edu`, `gov.xx`, `ac.xx`, and so on; the full list is `SUFFIX_RULES` in `categorize.mjs`), plus four government namespaces that are registrable domains themselves: `europa.eu`, `canada.ca`, `admin.ch`, `bund.de`.

The Public Suffix List is used only to reduce the host of a Wikidata website to its registrable domain.

Five labels: education, government, news, retail, community. A domain can carry several; the primary label follows that order. The class roots, the query text, the query time (2026-10-05), the result files and the version of the Public Suffix List are stored with the data, and `categorize.mjs --check` recomputes `categories.csv` from them.

Counts of labelled domains in the sample (S1 / S2 / S3 / S4): news 42 / 152 / 67 / 9; government 20 / 126 / 97 / 41; education 9 / 73 / 113 / 46; retail 25 / 74 / 33 / 18; community 46 / 33 / 7 / 1. 1,015 of the 10,000 domains carry a label. Coverage favours organisations that Wikidata documents well, and the labels contain errors that were seen during construction (a mail service labelled education; a chess site, a lyrics site and a classifieds site labelled news; a bank and a chamber of commerce labelled government). **News, government and education** are the categories the study reports on. Retail and community are too thin outside the top strata and are shown for description only.

For education and government the two sources can be set against each other. Education: 132 domains are labelled by both, 55 by Wikidata alone, 54 by a name rule alone. Government: 95 by both, 65 by Wikidata alone, 124 by a name rule alone. News, retail and community rest on Wikidata alone: the open directory that was planned as a second source (Curlie) could not be retrieved on 2026-10-05, and no other open list was substituted. Results by news category therefore depend on one source and are to be read with that in mind.

A category is reported as a rate in a stratum only when it has at least 30 sampled domains there; below that only counts are given.

**EU proxy.** A domain is "EU" when its top-level domain is the country code of one of the 27 member states or `.eu` (including their internationalised forms). This is a proxy for where a site is addressed, not for where its operator is established: an EU publisher under `.com` is not in it. 989 sampled domains are (34 / 227 / 288 / 440).

### 5.11 Home-page signals

The home page is requested once as HTML. From it the study records AI-related values of `<meta name="robots">` and `X-Robots-Tag`, TDM reservation, discovery links, inline RSL, and the `Content-Signal` and `Content-Usage` response headers. These describe one page per site and say nothing about the rest of it. They are reported as counts in an appendix table and support no headline claim.

---

## 6. Outcomes

Numbers are named by their key in `numbers.json`, the file from which the manuscript takes every figure. A key ends in a scope: `s1` to `s4` for a stratum, `top10k`, `top100k` and `top1m` for stratum-weighted estimates, `sample` for the unweighted pooled sample (Appendix C).

### 6.1 Primary outcomes

These are the quantities the abstract and the conclusions may lead with.

**P1. Refusal of four widely tracked tokens.** For GPTBot, CCBot, Google-Extended and ClaudeBot: the share of sites that fully disallow the token, under both counting rules and both denominators, per stratum and stratum-weighted. Example keys: `refusal.gptbot.explicit.robots.s1`, `refusal.gptbot.explicit.all.s1`, `refusal.gptbot.effective.robots.s1`, `refusal.gptbot.effective.all.s1`, `refusal.gptbot.explicit.robots.top10k`, `refusal.gptbot.explicit.robots.top1m`; likewise `refusal.ccbot.explicit.robots.s1`, `refusal.google-extended.explicit.robots.s1`, `refusal.claudebot.explicit.robots.s1`. Table 3; Figure 1.

**P2. Purpose differentiation.** The distribution of sites with robots.txt rules over the policy types, first-party scheme, effective rule, per stratum (for example `policy.strict.effective.open.robots.s1`, `policy.strict.effective.training-only.robots.s1`), the same on own text (`policy.strict.effective-own.training-only.robots.s1`), and tests C1 and C2. Table 5; Figure 2.

**P3. Authorship.** Among sites with robots.txt rules, the share whose file contains an intermediary block (`authorship.intermediary-only.robots.s1` plus `authorship.intermediary-plus-own.robots.s1`); among sites that explicitly refuse an AI token, the share that do so only inside an intermediary block (`authorship.ai-refusal.intermediary-only.s1`); and P1 recomputed on own text (`refusal.gptbot.explicit-own.robots.s1`). Table 6; Figure 3.

**P4. Invitation.** Valid files as a share of all sampled domains, per file and per stratum (for example `invite.llms-txt.strict.all.s1`); the share of sites with any valid invitation (`invite.any.strict.all.s1`); and how much a status-only count overstates (`invite.llms-txt.overcount`). Table 7; Figure 4.

**P5. Consistency.** Same-purpose conflicts as a share of the sites with a valid invitation, first-party scheme (`conflict.same-purpose.strict.invited.s1`, `conflict.same-purpose.strict.invited.sample`), with void invitations (`conflict.void.strict.invited.sample`) and co-occurrence (`conflict.co-occurrence.invited.sample`) beside it. Table 8.

**P6. Reachability.** The outcome of the research identity's robots.txt request over all sampled domains (`robots.research.rules.s1`, `robots.research.refused.s1`, and the other five groups); the discordant pairs (`pair.browser-only.s1`, `pair.research-only.s1`); and test C6. Tables 2 and 9; Figure 5.

### 6.2 Secondary outcomes

- The trend over rank strata (test C3) and the two category contrasts (tests C4, C5), with the category table (`category.news.training-explicit.s1`, `category.eu.training-explicit.s1`).
- Refusal by purpose class: at least one token, and every token, of each class refused (`purpose.strict.training.any.effective.robots.s1`, `purpose.strict.search-index.any.effective.robots.s1`, `purpose.strict.user-fetch.any.effective.robots.s1`), under both schemes (`purpose.community.training.any.effective.robots.s1`). Table 4.
- All 54 tokens under both rules, with named, partial and explicit-allow counts, pooled over the sample (Table A1); the sixteen tokens of Figure 1 by stratum (Table A7).
- Any AI token refused (`refusal.any-ai.explicit.robots.s1`), the `*` group (`refusal.star.full.robots.s1`), sites that refuse AI crawlers only through `*` (`refusal.star-only.robots.s1`), and the search-engine baseline (`refusal.baseline.any.effective.robots.s1`).
- Signals in robots.txt: presence, with and without intermediary text (`signal.content-signal.present.robots.s1`, `signal.content-signal.own.robots.s1`), stated values (`signal.content-signal.ai-train-no.robots.s1`), `License` records (`signal.license-line.present.robots.s1`), `Content-Usage` records (`signal.content-usage.present.robots.s1`); and Content-Signal `ai-train` against what the same file does to training crawlers (`consistency.cs-ai-train.yes.training-all.sample` and the other eight cells). Table A3.
- Invitation by stratum and by group (`invite.any.registered.strict.all.s1`, `invite.any.file.strict.all.s1`); an Agent Card at either path (`invite.a2a-either-path.strict.all.sample`); an MCP or agent-resource discovery file at any of the four probed paths (`invite.mcp-any-path.strict.all.sample`); conflicts per invitation (`conflict.by.llms-txt.same-purpose.strict.sample`) and under the community scheme (`conflict.same-purpose.community.invited.sample`).
- The same training-against-search and training-against-user comparison for the three other operators that document separate tokens (Anthropic, Mistral AI, Amazon). Descriptive; unadjusted p-values. Table A5.
- Collection facts: record states (`collection.collected.s1`), sites that answered at all (`collection.answered.s1`), stop reasons (`collection.stop.robots-unreachable.sample`), how sites treated the study's crawler (section 8), catch-all responders (`invite.soft-root.sample`, `invite.soft-well-known.sample`). Table 1.

### 6.3 Exploratory

Everything else in `results.json`, home-page signals (`home.meta-robots.noai.sample`, `home.tdm-reservation.sample`), and any analysis thought of after the data are seen. Such results are labelled exploratory or post hoc wherever they appear, and none of them is called a test.

---

## 7. Statistical analysis

### 7.1 Estimates in one stratum

The estimate is the count over the denominator. Every share is printed with its count and denominator. The 95% interval is Wilson's score interval; when the count or its complement is below 10 it is the exact (Clopper–Pearson) interval instead. (Brown, Cai and DasGupta, *Statistical Science* 2001, show the usual Wald interval to be unreliable; Wilson's is among the intervals they recommend.)

S1 is a census of ranks 1 to 1,000. For the list itself it has no sampling error. Its intervals are given under a super-population reading (the thousand sites as a draw from whatever produces the top of such a list), and each number carries a note that says so. The intervals of S2 to S4 do not apply a finite-population correction; for S2, where a third of the stratum was sampled, they are therefore somewhat wider than necessary.

### 7.2 Stratum-weighted estimates

For "all ranks down to 10,000", "to 100,000" and "to 1,000,000" the strata are combined with weights equal to their population sizes. Many quantities are ratios whose denominator is itself random (sites with robots.txt rules), so the estimator is a ratio of two estimated totals. With stratum *h* of size *N<sub>h</sub>*, sample *n<sub>h</sub>*, *d<sub>h</sub>* sampled sites in the denominator set and *k<sub>h</sub>* of those with the property:

```
R = Σ N_h (k_h / n_h)  /  Σ N_h (d_h / n_h)

Var(R) ≈ Σ N_h² (1 − n_h/N_h) s_h² / n_h  /  ( Σ N_h (d_h / n_h) )²

s_h² = sample variance, within stratum h, of  e = y − R·x
       (y = 1 for a site with the property and in the denominator set, x = 1 for a site in the denominator set)
```

This is the combined ratio estimator for a stratified sample, with its linearised variance and a finite-population correction (Cochran, *Sampling Techniques*, 3rd ed., chapter 6). The census stratum contributes no variance. The interval is R ± 1.96 standard errors, cut at 0 and 1. It is printed only when at least 10 sampled events and 10 sampled non-events stand behind the estimate; otherwise the estimate is printed without an interval and the counts are given. When every site is in the denominator set the formula reduces to the ordinary stratified proportion.

### 7.3 The pooled sample

Shares of the whole sample without weights describe the sample and estimate no population: S1 is one tenth of the sample and one thousandth of the list. They are used for totals in tables and for quantities too rare to estimate by stratum, and are always labelled as such.

### 7.4 Tests

Six comparisons are tests. Nothing else in the paper is.

| | Question | Sites | Outcome and groups | Test |
| --- | --- | --- | --- | --- |
| C1 | Do sites refuse OpenAI's training crawler more than its search crawler? | with robots.txt rules | GPTBot fully disallowed against OAI-SearchBot fully disallowed, effective rule, paired within site | exact McNemar |
| C2 | …more than its user-triggered fetcher? | with robots.txt rules | GPTBot against ChatGPT-User, effective rule, paired | exact McNemar |
| C3 | Does refusal of training crawlers change with rank? | with robots.txt rules | at least one `training` token explicitly fully disallowed, over strata S1 to S4 scored 1 to 4 | linear-by-linear association, M² = (n − 1)r², one degree of freedom |
| C4 | Do news sites refuse training crawlers more than other sites? | with robots.txt rules | the same outcome, news label against no news label, stratified by rank stratum | Cochran–Mantel–Haenszel, no continuity correction |
| C5 | Do sites under EU top-level domains differ from the rest? | with robots.txt rules | the same outcome, EU proxy against the rest, stratified by rank stratum | Cochran–Mantel–Haenszel, no continuity correction |
| C6 | Is the policy file answered more often for a browser string than for the declared research identity? | collected | robots.txt request answered, browser identity against research identity, paired within site | exact McNemar |

All tests are two-sided. The family-wise error rate is 0.05, split evenly (Bonferroni): each test is judged at 0.05 / 6 = 0.0083 (`test.alpha`, `test.family-size`). Each result is reported with its exact p-value (`test.c1.p`, `test.c6.p`) and with the counts behind it, significant or not: the discordant pairs for the paired tests (`test.c1.first-only`, `test.c1.second-only`, `test.c1.pairs`), the statistic for C3 (`test.c3.statistic`), and for C4 and C5 the Mantel–Haenszel common odds ratio with the Robins–Breslow–Greenland interval (`test.c4.odds-ratio`). Table 10.

The paired tests and the trend test are computed on the sample without weights. They ask whether the difference exists among the sampled sites; the per-stratum values (Figure 1 for C1 and C2, Table 10 for C3, Table 9 for C6) show whether its direction holds in each stratum. The exact McNemar test needs at least eight discordant pairs, all in one direction, to fall below 0.0083. In C4 and C5 a stratum that has no site in one of the two groups carries no information and is left out; the table lists the counts of every stratum.

**What earlier work leads one to expect.** C1, C2: agent-level reports (Bouchaud and Ramaciotti; Cloudflare Radar; AI Visibility) find search and user-triggered agents refused less than training agents. C3: refusal falls with rank in every study that reports rank buckets. C4: news sites sit far above other sites in three independent sources. C5: no expectation is registered. C6: Liu et al. found pages blocked for an AI crawler's string and served to a browser string on 14% of the top 10,000; the present test concerns the policy file and a research identity, for which no earlier figure was found.

The implementations of the interval and test procedures were checked against worked examples from the literature and against values computed by hand; the test file `test/research/wmrw-analyze.test.mjs` lists them.

### 7.5 Small numbers

Many invitation files will have single-digit counts. For those the paper gives the count and the exact interval and makes no comparison between strata or categories. No category is compared in a stratum where it has fewer than 30 sampled domains.

---

## 8. Missing data, exclusions and opt-outs

- **Nothing is imputed.** A request that was not sent or not answered is unknown and is reported as such.
- **No site is replaced.** Unreachable, refusing and invalid domains stay in the sample and in "all sampled domains".
- **Opt-outs before or during collection.** A site that disallows the crawler in robots.txt receives the two robots.txt requests and nothing else. It stays in the dataset, because that file is the object of study. The paper reports how many sites name the crawler in a group (`collection.opted-out-by-robots.sample`) and how many disallowed at least one of its requests, by name or through `*` (`collection.disallowed-some.sample`).
- **Opt-outs by message.** A domain whose operator asks to be left out is never contacted, or, when the request arrives later, its record is removed. It is counted as `excluded` in its stratum (`collection.excluded.sample`), stays in "all sampled domains" as not collected, and has no row in the published per-site table.
- **Collector defects.** A site stopped by a defect is `collector-error`; it is not requested again.
- **Interrupted sites.** A site that a crashed run had started is not requested again automatically: the study page promises one pass. Such sites are `no-record` and are listed in the run manifest.

---

## 9. Sensitivity analyses

Registered in advance; each is reported whatever it shows.

1. **Own text.** Every refusal figure and policy type without intermediary blocks (5.6).
2. **Counting rule and denominator.** The four cells of 5.3 for every token.
3. **Purpose scheme.** Purposes, policy types and conflicts under the community scheme.
4. **Google's reading.** The explicit figures of the four headline tokens when the file is read as Google's parser reads it (`sensitivity.google-reading.gptbot.explicit.robots.sample`), and the share of files where any verdict changes under that reading or under three narrower ones (`sensitivity.reading-differs.google.sample`).
5. **Stricter full block.** Every path disallowed and no `Allow` rule in the group (`sensitivity.strict-full.gptbot.explicit.robots.sample`).
6. **Either identity's file.** The explicit figures when the file of whichever identity obtained rules is used (`sensitivity.best-file.gptbot.explicit.best-robots.sample`), and how many sites that adds (`sensitivity.best-file.gain.sample`, `pair.rules.browser-only.sample`).
7. **Answered-only denominator for files.** Adoption among sites where the request for the file received an HTTP answer (`invite.llms-txt.strict.observed.s1`).
8. **Lenient tier and status-200 counts** for every file (`invite.llms-txt.lenient.all.s1`, `invite.llms-txt.status200.all.s1`).
9. **Redirected robots.txt** (5.2, point 2) and **soft 404** (`robots.research.detail.soft-404.sample`): how many sites each rule moves out of "rules".
10. **Order of the paired requests** (5.9).
11. **Classic-search signals against the search-engine baseline** (`conflict.search-signal.all.sample`).

---

## 10. The pilot, and what may change after it

The pilot collects 200 domains with the frozen instrument. Its purpose is to find defects before 9,800 more sites are contacted.

**May change after the pilot, each change logged in section 11:**

- defects of the collector that make it request more, or more often, than the study page allows, or that corrupt what it records;
- run parameters that do not alter what is requested of a site: concurrency, the location of the data directory;
- additions to the catalogue of edge-page signatures and of robots.txt fingerprints, and the raising of an evidence level, under the rule below;
- defects of the analysis code, when a definition in this document is implemented wrongly. The definition stands; the code is corrected.

**May not change:** the sample; the list of probes; the two identities; the request limits; the definitions of section 5; the primary outcomes; the six tests and their threshold.

**Sites of the pilot are not contacted again.** Their records enter the final dataset. If a defect found in the pilot affected what was recorded for them, the affected fields are recomputed from the stored bodies where that is possible; where it is not, the pilot records are left out of the affected measures, and the log says how many.

**Rule for verifying a fingerprint or signature.** An unverified fingerprint or edge signature can be raised to verified only when (a) a source for the text is cited (the vendor's documentation or the published default file), and (b) a hand check of matching files collected in this study confirms it: all matching files when there are 20 or fewer, otherwise a random 20, with at least 19 of 20 confirmed. The check and its outcome are logged. Fingerprints that stay unverified stay in `unverified-template`. No fingerprint is added or verified after the full results have been looked at by category or stratum.

---

## 11. Deviations log

Empty at the time of freezing. Every later change to a frozen input, to the collector, or to the analysis is entered here before the paper is written.

| Date | What changed | Why | Effect on results |
| --- | --- | --- | --- |
| 2026-10-09 | Before any collection: the link in the research User-Agent now points at the site's contact page (`https://maxsong.me/contact/`) instead of the study page. `config.mjs`, `codebook.json` and `frame.json` were regenerated; the study page shows the string actually sent. | The study page could not be deployed before the run (the deploy host was unreachable from the collection machine) and the author chose to proceed; a User-Agent must not advertise an address that answers 404 during the run. The contact page was live throughout. | None on measurements: the product token, every limit and every request are unchanged. Operators who followed the link during the run saw a contact form rather than this description. |
| 2026-10-09 | After 1,200 of the 10,000 sites had been collected: `readRecords()` in `collect.mjs`, which a resumed session calls at start-up to learn which sites already have a record, split lines with Node's `readline`, which also ends a line at U+2028 and U+2029. One record quoted a U+2028 from a site's content (`JSON.stringify` leaves it unescaped), so the second full session failed at start-up with a JSON parse error before contacting any site. The reader now splits at LF only (`splitLines`); the writer, the on-disk format and the 1,200 existing records are unchanged, and all of them parse. `SCHEMA.md` now states the LF-only rule and a regression test covers it. | A defect in the reader, found when the second full session was started. | None on results: no measurement, request or definition changed, and no site was requested twice (the failed session stopped before its first request and left no manifest entry). The run manifest records the new code hash of `collect.mjs` from the next session on. |
| 2026-10-09 | A tool for the fingerprint rule of section 10 was added (`code/tools/fingerprint-review.mjs`, with its test): it lists every collected file that matched an unverified fingerprint or edge signature and, when there are more than 20, draws the random 20 with a recorded seed (default: the study id), so that the draw can be repeated. It reads the records only. | The rule asks for a random 20; the draw has to be reproducible and documented. | None on results. |
| 2026-10-09 | While the collection was still running (4,200 of 10,000 records written), the derive, analyze, figures and page-check steps were rehearsed on those records, with the outputs written outside the repository and discarded. In doing so the author saw the rehearsal's headline refusal table by stratum and its comparator table. Section 10 says that no fingerprint is added or verified after the full results have been looked at by category or stratum; the rehearsal was partial and is recorded here so that a reader can weigh it. No fingerprint has been added since freezing; verification follows the pre-registered rule (a cited source, and 19 of 20 hand-checked matches) and is carried out on the complete records before the final analysis is run. | To find defects in the analysis tools before the collection ended, rather than after. | None on measurements. The reader can judge whether the rehearsal could have influenced the fingerprint checks; the checks themselves are logged below with their samples. |
| 2026-10-10 | After the collection and before the final analysis, the rule of section 10 was applied to the 28 unverified entries (sources: a workflow of 8 finders and 8 refuters; hand check: 8 checkers and 8 refuters over 366 drawn files, a file counting as confirmed only when both agreed). Five robots.txt fingerprints (`yoast-block`, `wordpress-core-rules`, `drupal-default`, `prestashop-generated`, `joomla-default`) and eight edge signatures (`vercel-challenge-header`, `vercel-deny-header`, `aws-waf-action-header`, `cloudfront-request-blocked`, `akamai-access-denied`, `datadome-captcha`, `f5-request-rejected`, `anubis-challenge`) were raised to `vendor-documented`; each entry now records its `sourceUrl`, `verifiedOn` and `handCheck`, and `codebook.json` was regenerated (new sha256 in section 2; `config.mjs` unchanged). The other 15 stay unverified: no vendor source although all, or all but one, of the drawn files were confirmed (`shopify-default` 19 of 20, `squarespace-default` 9 of 9, `wix-auto-generated` 13 of 13, `wordpress-com-firehose` 4 of 4); hand check below the threshold (`blogger-default` 3 of 20, `imperva-incident` 6 of 20, `rails-default` 2 of 20, `knownagents-per-agent` 0 of 20, `perimeterx-captcha` 18 of 20 with 2 files whose markers lie beyond the stored body head, `ghost-default` 2 of 3); or no matching file (5 entries). The description of `wordpress-core-rules` now states that a match means the file carries WordPress's two default rules (all 20 checked files do; 5, by both reviewers, are core output as served), which is the reading of class 4 in section 5.6; the descriptions of `yoast-block`, `drupal-default`, `prestashop-generated`, `joomla-default` and `cloudfront-request-blocked` gained a clause on what the match does and does not show. Full log with every count and the interpretations applied: `implementation/fingerprint-verification.md`. | Section 10 requires the check before the final analysis. | Files that match a raised fingerprint are classed `platform-template` instead of `unverified-template`; refusals recognised by a raised signature count in `robots.research.refused-verified-edge`. Only evidence levels, source links and description texts changed; no pattern, test or code path. |
| 2026-10-10 | Found by the hand check and **not corrected**: `imperva-incident` matches the `_Incapsula_Resource` script path that Imperva injects into the pages it proxies, and robots.txt lines that mention that path, at any status and content type. Over the complete records it matched 130 responses from 20 domains; ten are the real robots.txt of six sites (status 200, `text/plain`), recorded as `edge-block` with their rules unparsed, and two more are 404 pages recorded as `edge-block` instead of `absent`. Three of these eight sites were left alone after the home page was also classed as refused. The signature stays `implementer-recall`; the collector, the records and the analysis are unchanged. | Correcting a signature and recomputing stored outcomes after the results have been seen is the author's decision; the defect is logged so that the paper can disclose it and a reader can recompute. | If corrected: at most six sites move from "robots.txt refused" to "obtained" and two from refused to absent (0.08% of the sample); the three stopped sites cannot be completed without new requests. The paper reports the defect with these counts. |
| 2026-10-10 | After the manuscript was written, two edits to `analyze.mjs` (frozen analysis code), both found by the closing review. First, the caption of the home-page signals table (A6): an edit made the same day while the page was integrated had labelled the table "not pre-registered, see protocol section 7", which names the wrong section and contradicts section 6.3; the caption now reads "Exploratory, as registered in protocol section 6.3". Second, table cells are escaped before they are written as Markdown, because the engine passed text such as `<meta name="robots">` through as an HTML element and four rows of A6 lost that text in the built page and PDF. | A caption error and a rendering defect in the built outputs. | None on any number: the measures, the counts and `results.json` are unchanged; only the text of `tables.json` and the built page change. |
| 2026-10-10 | Two `handCheck` strings in the codebook were corrected after the closing review, which changes the codebook's hash (section 2) and nothing else. `wordpress-core-rules` had recorded that 7 of the 20 drawn files were core output as served, which was the checker's count alone; the count on which both reviewers agreed, and which the verification log records, is 5. `cloudfront-request-blocked` now also records that both reviewers agreed on 16 of the 20 files (the 4 disputed: 3 of the "Bad request." variant and 1 page generated by CloudFront but relayed by another vendor's edge); the verification log's entry for it, which had called all 4 disputed files the "Bad request." variant, was corrected to match. `codebook.json` was regenerated; the manuscript uses the same counts. | Counts in the codebook, the log and the manuscript did not agree. | None on any number: no pattern, test or evidence level changed; `results.json` and `sites.meta.json` record the new hash. |
| 2026-10-10 | After the closing review: the raise of `cloudfront-request-blocked` recorded above was withdrawn the same day. The hand check's own rule counts a file as confirmed only when both reviewers agreed; they agreed on 16 of 20 (3 "Bad request." pages from a misconfigured distribution and 1 page relayed by another vendor's edge were disputed), below the 19 required, and the raise had rested on a reading written in Part (a) of the log rather than on that count. The entry is back at `implementer-recall`, keeping its source and hand-check counts for the record; `codebook.json` was regenerated (new sha256 in section 2). | The closing review judged the raise an interpretation, not the mechanical rule of section 10. | Its matches still count as refusals; the policy-file refusals among them leave `robots.research.refused-verified-edge` and are reported as unidentified. Seven edge signatures, not eight, are raised; the manuscript's counts changed accordingly. |
| 2026-10-10 | `analyze.mjs` gained keys for counts that the manuscript names and that had no key: `robots.research.refused-verified-edge-vendor.<vendor>` (policy-file refusals by a verified signature, per vendor), `robots.research.refused-unverified-edge` (by an unverified signature) and `refusal.any-ai.explicit.via-builder-default` (an exploratory count: sites that explicitly refuse an AI token and carry the default text of one of the two hosted builders whose defaults name AI tokens, `squarespace-default` or `wix-auto-generated`, both unverified fingerprints); `loadCodebook()` in `derive.mjs` now also returns the vendor of each edge signature, which the first group needs. | The manuscript's rule that every printed number is a key of `numbers.json`; the closing review found places that printed counts read off the per-site table instead. | None on existing numbers: only new keys are added; the per-site table is unchanged. |

---

## 12. Limitations known in advance

1. **One vantage point, one pass, one day per site.** A site can answer differently by country, by network and by hour. Nothing here measures change over time.
2. **robots.txt is a statement, not an enforcement.** The study measures what sites say. It does not measure whether crawlers comply, nor blocking that happens in firewalls without a word in robots.txt, which earlier work found to be common.
3. **The browser identity is a header.** Section 5.9.
4. **The list is of domains, not of websites.** Many entries serve no web content. Rates over all sampled domains are lower than the same rates over a list of websites, which is why every rate has a second denominator.
5. **Registrable domain and root paths only.** Policy files on subdomains, path-suffixed OAuth metadata and `llms.txt` files below the root are not probed.
6. **Conflicts hidden behind a `*` rule are not observable** by a crawler that obeys robots.txt. Section 5.8.
7. **Purposes are what operators declare.** A token labelled "search" may feed other uses. Twenty-four AI tokens have no usable declaration.
8. **Google's AI-search control is invisible from outside.** Section 5.4.
9. **Authorship is recognised only for known text.** One intermediary's blocks are verified; other vendors that write robots.txt, and templates not in the catalogue, fall into the residual class.
10. **Edge pages of vendors other than Cloudflare are not verified.** Section 5.2.
11. **Categories cover a tenth of the sample and contain errors.** Section 5.10. The EU proxy is a top-level domain, not a jurisdiction.
12. **Validation rules are minimal.** "Valid" means a file passes a small structural test taken from its specification, not that it conforms to it or that anything consumes it.
13. **The specifications are moving.** Several probed paths are drafts or vendor conventions that may be renamed. The codebook records the status of each on 2026-10-04.
14. **A self-identified research crawler may be treated unlike any crawler a site has heard of.** RQ5 measures exactly this, and it also conditions RQ3: files that a site serves only to known agents are missed.

---

## 13. Ethics

The study has had no institutional review. It was designed against the principles of the Menlo Report (Dittrich and Kenneally, 2012) and answers the four questions that Partridge and Allman (*Communications of the ACM* 59(10), 2016) ask of a measurement paper.

**Respect for persons.** The subjects are websites, reached through their public policy files. Operators are told who is asking and why: every request carries a `User-Agent` that names the study and links to a page that states its purpose, the full list of requests, the limits, and two ways to opt out. One request per site carries a browser string instead; the page discloses it and the reason for it. The crawler never uses the name of another organisation's bot.

**Beneficence.** Load is the main risk and is bounded by construction: at most 26 `GET` requests per site, in one pass, at most one per second per site, bodies cut at 1 MB, at most one retry, no request after a site signals overload or refuses the crawler. The files requested are ones that sites publish for machines to read. The benefit is an open account of how access rules for AI are actually stated.

**Justice.** Sites are selected by rank and a public hash, not by who they are. The paper's text names no individual site as refusing, inviting or contradicting itself.

**Respect for law and public interest.** The crawler obeys robots.txt, sends no credentials, attempts no challenge and circumvents nothing. Only `GET` is used. Code and data are published so that the numbers can be checked.

**Partridge and Allman's questions.**

1. *Could collection harm a person?* The requests are few, slow and addressed to machine-readable public files. No form is submitted, nothing is written to any site.
2. *If data are reused, is their collection documented?* The Tranco list, the public suffix list and Wikidata are public datasets used under their terms; their versions and retrieval times are recorded.
3. *Can the data reveal private information?* The collector seeks none. It stores status codes, a fixed list of response headers, and the bodies of policy and discovery files. Cookies are never stored. Home pages are kept only as extracted signals and a hash. A `security.txt` contains a contact address by design; the published per-site table contains derived fields only and no file bodies.
4. *Anything else?* The paired request sends a browser string from a client that is not a browser. It is one request per site for a file meant for crawlers, it is disclosed to operators in advance, and it is the only way to learn whether a declared crawler is shown the same policy as everyone else.

**Promises of the study page, and where they are kept.** Identity, 26 URLs, `GET` only, five redirects, one request per second, 1 MB, one retry, robots.txt obeyed, the single browser-string exception, no cookies kept: all are constants of `code/config.mjs`, and a test compares that file with the page. Opt-out by robots.txt and by message: section 8.

The number of sites that named the crawler in robots.txt, the number that asked by message to be left out, and any complaint received will be reported.

---

## 14. Comparability with other measurements

No two published series share a frame, a denominator and a counting rule. The paper places its figures beside others in a table that states all three for each row, and draws no trend line through rows that differ in any of them.

| Source | Frame | Counting rule | Denominator | This study's matching cell |
| --- | --- | --- | --- | --- |
| Liu et al. (IMC '25), October 2024 | Tranco domains present in every monthly top 100,000 and with a robots.txt in every snapshot | explicit full disallow; any of 24 agents | sites with a robots.txt | any AI token, explicit rule, sites with rules, weighted to the top 100,000. The frame differs (no survivorship filter here) and so does the token list (24 against 52) |
| Cloudflare Radar, weekly | Radar's top 10,000 domains | "fully disallowed" per agent | files parsed | explicit rule, sites with rules, top 10,000 |
| AI Visibility, 1 October 2026 | Tranco top 1,000 plus a UK list | effective, wildcard inherited | all domains reached | effective rule, all sampled domains, S1. The denominators differ slightly (reached against sampled) |
| Bouchaud and Ramaciotti, 2025 | Chrome UX Report top million | "fully blocked"; denominator not stated explicitly | — | reported beside both denominators |
| Web Almanac 2024, 2025 | Chrome UX Report origins | agent *named* in the file | all sites | the "named" count of Table A1 |

A same-month row from Cloudflare Radar (the scan dated closest to the collection) will be retrieved by hand when the collector runs, stored with its retrieval time in a small file of the form `{ "schema": "wmrw-comparators/1", "rows": [...] }`, and printed beside this study's top-10,000 estimate. It is a comparator, not a validation: the frames, the clients and the parsers differ.

---

## 15. Decisions to confirm at freezing

Each has a default, which is what the code does today. If the author confirms the defaults, this section is replaced by a line that says so.

*Confirmed by the author on 2026-10-09: all seven defaults below are adopted as written.*

1. **robots.txt behind a path-changing redirect** is counted as absent (5.2). The alternative, closer to RFC 9309, is to accept a redirected body that is structurally a robots.txt. Default: keep the rule that guards every probe alike, and report the number of sites it affects.
2. **The four government namespaces** added by hand (5.10). Default: keep them. They are listed in the code, and in this sample they decide two labels (`europa.eu` and `canada.ca`; the other two are not in the sample).
3. **Retail and community** stay as descriptive labels only (5.10).
4. **The verification rule for fingerprints** (section 10): a cited source, and 19 of 20 hand-checked matches.
5. **Pilot records** are kept and treated as section 10 says.
6. **The per-site table** published with the paper contains derived fields only, for every sampled domain except those excluded at an operator's request. `sample.csv` stays complete, because it can be recomputed by anyone from the public list and the public salt.
7. **The six tests and the six primary outcomes** of sections 6 and 7 are the registered set.

---

## 16. Outputs and reproducibility

| Output | Produced by | Content |
| --- | --- | --- |
| `sites.csv`, `sites.meta.json` | `derive.mjs` | one row per sampled domain that may be published, with every derived variable; counts per record state and stratum |
| `results.json` | `analyze.mjs` | every measure in every scope, the tests, the definitions in use |
| `numbers.json` | `analyze.mjs` | the flat map from which the manuscript takes each number, with numerator, denominator, interval and a note |
| `tables.json` | `analyze.mjs` | the tables of the paper |
| `figures/*.svg`, `figures/figures.json` | `figures.mjs` | the figures, drawn from `results.json` alone |

`analyze.mjs` gives byte-identical output from the collector's records and from the published per-site table, and `--check` on each script requires the files on disk to equal a fresh computation. The manuscript cannot print a number that is not in `numbers.json`; the page builder fails on an unknown key.

All analysis code was developed and tested on synthetic data in which the true value of every quantity is known (`synthetic.mjs`), and on which a test requires each number to be recovered exactly. Output computed from synthetic or test data is stamped as such in every note, caption and figure, and the scripts refuse to write it into the paper's folder.

The raw records stay outside the repository. The paper publishes the per-site table, the code, the sample, the frame record, the category sources, this protocol and its log.

---

## Appendix A. The 26 requests and their rules

"Present" and "valid" apply after the generic rejections of section 4. The full statements are in `codebook.json` (`probes[].validatorRules`).

| # | Request | Group | Present (lenient) | Valid (strict) |
| ---: | --- | --- | --- | --- |
| 1 | `/robots.txt`, research identity | — | 200, not HTML, not empty | at least one line with a robots.txt field |
| 2 | `/robots.txt`, browser identity | — | as 1 | as 1 |
| 3 | `/<16 random hex digits>.txt` | control | — | — |
| 4 | `/.well-known/<16 random hex digits>` | control | — | — |
| 5 | `/`, `Accept: text/html` | home page | 200, not empty | — |
| 6 | `/`, `Accept: text/markdown` | proposal | 200 and media type `text/markdown` | and the body is neither HTML nor empty |
| 7 | `/llms.txt` | proposal | 200, not HTML, not empty | first non-blank line is a level-one Markdown heading |
| 8 | `/llms-full.txt` | proposal | as 7 | at least 200 bytes and not identical to the site's `/llms.txt` |
| 9 | `/ai.txt` | proposal | as 7 | no strict rule |
| 10 | `/.well-known/api-catalog` | registered | 200, parses as JSON | an object with an array `linkset` whose every member has a string `anchor` |
| 11 | `/.well-known/agent-card.json` | registered | 200, a JSON object | string `name`, array `skills`, and array `supportedInterfaces` or string `url` |
| 12 | `/.well-known/agent.json` | proposal | as 11 | as 11 |
| 13 | `/.well-known/mcp/server-card.json` | proposal | 200, a JSON object | an object `serverInfo`, or a `transport`, `endpoint` or `url` member |
| 14 | `/.well-known/mcp.json` | proposal | as 13 | as 13 |
| 15 | `/.well-known/ai-catalog.json` | proposal | 200, a JSON object | a non-empty array `entries` whose every entry has `identifier` |
| 16 | `/.well-known/ard.json` | proposal | 200, a JSON object | an array `entries` |
| 17 | `/.well-known/agent-skills/index.json` | proposal | 200, a JSON object | an array `skills` whose every entry has a string `name` |
| 18 | `/.well-known/agents.json` | proposal | 200, a JSON object | no strict rule |
| 19 | `/.well-known/ai-plugin.json` | proposal | 200, a JSON object | no strict rule |
| 20 | `/auth.md` | proposal | 200, not HTML, not empty | a level-one heading that contains `auth.md` |
| 21 | `/.well-known/oauth-protected-resource` | infrastructure | 200, a JSON object | a string `resource` |
| 22 | `/.well-known/oauth-authorization-server` | infrastructure | 200, a JSON object | a string `issuer` that begins with `https://` |
| 23 | `/.well-known/openid-configuration` | infrastructure | 200, a JSON object | as 22 |
| 24 | `/.well-known/tdmrep.json` | proposal | 200, parses as JSON | an array with at least one object that has `location` and `tdm-reservation` 0 or 1 |
| 25 | `/.well-known/security.txt` | infrastructure | 200, not HTML, not empty | a `Contact:` line |
| 26 | the licence document named by a `License:` line of the site's robots.txt, if any, when it is on a host of the sampled domain or on the host that served robots.txt | proposal | 200, looks like XML | root element `rsl` in the RSL namespace, with a `license` element |

The two robots.txt requests come first, in an order fixed per domain by one bit of a hash. Requests 3 to 25 follow in an order fixed per domain by a hash; when robots.txt was refused, the home page is asked for next, to decide whether the site is left alone. Request 26 comes last and is sent only when robots.txt points to a licence document.

## Appendix B. Token sets

**First-party scheme.**

- `training` (7): GPTBot, ClaudeBot, Applebot-Extended, Amazonbot, MistralAI-Training, AI2Bot, webzio-extended.
- `search-index` (11): OAI-SearchBot, Claude-SearchBot, meta-webindexer, Amzn-SearchBot, PerplexityBot, DuckAssistBot, MistralAI-Index, ImagesiftBot, YouBot, PetalBot, Diffbot.
- `user-fetch` (10): ChatGPT-User, Claude-User, Google-Agent, Google-GeminiNotebook, Google-NotebookLM, meta-externalfetcher, Amzn-User, Perplexity-User, MistralAI-User, Diffbot-User.
- `other` (24): OAI-AdsBot, anthropic-ai, Claude-Web, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, Google-CloudVertexBot, Applebot, meta-externalagent, meta-externalads, facebookexternalhit, FacebookBot, CCBot, webzio, omgili, omgilibot, Coherebot, cohere-ai, cohere-training-data-crawler, Bytespider, DeepSeekBot, xAI-SearchBot, Timpibot.

**Community scheme.**

- `training` (18): GPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, GoogleOther, Google-CloudVertexBot, Applebot-Extended, meta-externalagent, FacebookBot, Amazonbot, CCBot, MistralAI-Training, omgili, PetalBot, Diffbot, cohere-ai, Bytespider.
- `search-index` (4): OAI-SearchBot, Claude-SearchBot, Applebot, PerplexityBot.
- `user-fetch` (6): ChatGPT-User, Claude-User, meta-externalfetcher, Perplexity-User, DuckAssistBot, MistralAI-User.
- `other` (24): the remaining AI tokens.

**Baseline, in no AI class:** Googlebot, bingbot.

The paper's headline table uses GPTBot, CCBot, Google-Extended and ClaudeBot because these four appear in most earlier series. Two of them (CCBot, Google-Extended) are in the class `other` under the first-party scheme: Common Crawl's documentation does not state a use within the three classes, and Google-Extended is a control token for more than one use.

## Appendix C. How numbers are named

A key of `numbers.json` is `<measure>.<scope>`.

| Scope | Meaning | Carries |
| --- | --- | --- |
| `s1`, `s2`, `s3`, `s4` | one rank stratum | count, denominator, 95% interval |
| `top10k`, `top100k`, `top1m` | all ranks down to the bound, stratum-weighted (7.2) | estimate; interval when at least 10 events and 10 non-events were sampled |
| `sample` | the whole sample without weights (7.3) | count, denominator |

Measure families:

| Family | Content |
| --- | --- |
| `frame.*`, `count.*` | sizes of the list, the sample and the main denominators |
| `collection.*` | record states, stop reasons, how sites treated the study's crawler |
| `robots.<identity>.*` | outcome of each robots.txt request (5.2) |
| `refusal.<token>.*`, `refusal.any-ai.*`, `refusal.star.*` | the refusal layer, token by token and in aggregate (5.3) |
| `purpose.<scheme>.<class>.*` | refusal by purpose class (5.4) |
| `policy.<scheme>.<rule>.<type>.*` | policy types (5.5) |
| `authorship.*` | who wrote the file (5.6) |
| `invite.<file>.*`, `invite.any.*` | the invitation layer (5.7) |
| `signal.*`, `consistency.*` | signals inside robots.txt (5.7) |
| `conflict.*` | invitation against refusal (5.8) |
| `pair.*` | the paired comparison (5.9) |
| `category.<label>.*` | categories and the EU proxy (5.10) |
| `home.*` | home-page signals (5.11) |
| `sensitivity.*` | section 9 |
| `test.*` | the six tests (7.4) |

A suffix `-own` on a rule means "intermediary blocks removed". The denominators appear in the key as `robots` (sites with robots.txt rules), `all` (all sampled domains), `file`, `invited`, `observed` and `best-robots`.