How we check, in detail

What our bot reads

Once a day, for each of the 170 tracked sites, WalledWebBot reads up to six files: /robots.txt; /llms.txt, /ai.txt and /.well-known/tdmrep.json; the homepage, of which it keeps only the response headers and the <head>; and the Terms of Service page. With redirects and a www fallback that is at most about eight requests. It identifies itself honestly, obeys robots.txt for its own name, sends no cookies, runs no JavaScript, and never retries more than once. Across all sites it makes about one request per second.

How robots.txt is read

robots.txt is parsed under RFC 9309. Rules are matched to each agent's product token without regard to case. All groups that name an agent are merged; if none names it, the User-agent: * group applies. The longest matching path wins, and on a tie, Allow wins. Common typos such as Dissallow are read the way Google reads them, and flagged.

For each agent we test two paths: the homepage, and a path no site lists, which stands for "everything else". If both are disallowed, the agent is blocked from the site. If only one is (a site that allows just its homepage, say), it is partly blocked. Otherwise it is allowed. Rules for search results, carts, checkout and account pages are shown on each site page but do not change the label, because most sites keep every crawler out of those pages as routine hygiene.

The parser is tested against RFC 9309's rules and compared, answer by answer, with an independent open-source parser on a corpus of real files. Raw files are stored, so a parser fix can be re-run on the whole history.

Three kinds of AI bot

A block means different things for different bots, so each site shows three columns:

• AI training bots collect pages to train AI models. Blocking them keeps pages out of future training, but does not stop AI from reading the site to answer questions.
• AI search bots fetch pages so an AI product can quote and link to them.
• AI assistants fetch a page because a person asked an AI to, right now. 5 of the 7 AI assistants we track come from companies that say they may ignore robots.txt when a person asks for a page (OpenAI, Perplexity, Google, Meta, Amazon).

The list of bots is a reviewed config file. Each token was checked against its operator's documentation on 19 Oct 2026. It is reviewed monthly.

How the overall result is set

1. Charges AI wins over every other result when there is a verified paid signal: an HTTP 402 response to our bot, or a sourced statement that AI companies must license access.
2. Blocks AI requires one of four things: robots.txt blocks all 12 major agents (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User); a human-reviewed Terms of Service prohibition; at least two independent, verified community reports of a hard block; or, when robots.txt cannot be read, a sourced company statement that it blocks AI agents together with a refusal at the door (see the next rule).
3. Mixed signals produce Blocks some, never Allows AI. That includes a robots.txt that permits AI agents while the homepage turns away or challenges our bot, which says who it is, and a permissive robots.txt at a company that says, on the record and with a source we link, that it blocks or restricts AI agents (for example, at login).
4. A company statement can settle an unreadable site. If robots.txt cannot be read, the homepage turned our bot away, and the company says on the record that it blocks AI agents, the result is Blocks AI. A statement alone never sets Blocks AI.
5. When robots.txt cannot be read, a recent copy (three months at most) from the Internet Archive may stand in, and is labeled. Otherwise the result is Unknown. A refusal to our bot is recorded as evidence, but a refusal from one network location is not proof that a site blocks all agents.

Sites that let one company in by name

A site is listed under sites that let one or two AI companies in when its robots.txt names at least one crawler of an AI company and allows it, blocks none of that company's agents, and blocks agents from at least three other companies. Only agents a file names count, so a company simply missing from the file does not put a site on this list. A business agreement between the site and the named company is shown only when a primary source or reputable report confirms it; the Index never infers one from robots.txt.

The front door test

Our bot requests each homepage once, as itself, from the same place each day. We record whether the site answered normally, refused (401, 403, 429), served a challenge page, or asked for payment (402). The current runs come from a residential connection in the United States. Bot-management products often treat cloud-hosting addresses more harshly than home connections, so results are compared only between runs from the same place, and a change of location is never reported as a change by a site.

History from the Internet Archive

To show change since 2023 without waiting years for our own record, we read the Internet Archive's copies of each robots.txt: the first successful capture in each month, from Jan 2023 on. Identical copies are stored once. A copy stands for up to three months if no newer one exists. Some sites, for example Yelp and Booking.com, have asked the Archive not to serve their pages, so their history starts with our first check.

Archive captures sometimes alternate between two versions of a file served by different machines. A change is only recorded when the new version holds at the next capture too, or is the newest one. Changes from the Archive are dated by month and show the window in which they happened.

Reports from readers

Our bot sees only public signals. What an agent meets mid-checkout comes from people. Every report is held for moderation. A moderator marks it verified when there is evidence they can check, plausible when it is credible but unchecked, or rejected. Plausible reports are shown in a lighter style and never change a status on their own. Reports must not contain personal or health information. Contact emails are optional, stored apart from reports, seen only by moderators, and never published.

What we cannot see

• Anything behind a login. Our bot never logs in, so account-level blocks like those described by banks are outside its view.
• Whether an agent obeys the rules. robots.txt is a request. Several AI companies say their assistants may not follow it.
• Agents that do not identify themselves. Browser-based agents use the person's own browser, and some hosted agents use a generic browser identity. No robots.txt rule can address them by name.
• Google's AI Overviews and AI Mode. Google-Extended controls Gemini training and grounding, not those features, which follow Googlebot rules and a Search Console setting.
• Legal meaning. Terms of Service sentences are selected by keyword and quoted briefly with a link. They are not a legal reading.

Corrections and opt-out

Each site page has a "Report a mistake" link. Site owners can ask for a correction, and corrections are published in the change log. Owners can stop our bot with one robots.txt line, or ask not to be checked at all; we honor that within a week and list the site as "Owner requested no checking".

Legal and ethical notes

These notes describe how this project works. They are not legal advice.

• Respect for site rules. Our bot obeys robots.txt for its own name, identifies itself, and reads only what a site publishes for any visitor.
• No circumvention. No challenge solving, no browser fingerprinting, no proxies, no logins. If our bot is blocked, we record that and stop.
• Plain description. Results describe what anyone can observe, with evidence and dates. We avoid guessing at intent.
• Quotation. Terms of Service are hashed to detect changes; only short excerpts are quoted, with a link and a date.
• Independence. We take no money from tracked sites or AI companies.