WalledWebBot

What it fetches

At most once per day, for each tracked site, and never more than about one request per second across all sites combined:

1. /robots.txt, read under RFC 9309, the Robots Exclusion Protocol.
2. /llms.txt, /ai.txt and /.well-known/tdmrep.json, if robots.txt allows it.
3. The homepage, if robots.txt allows it. It keeps the response headers and the <head> element only, to read meta directives. It does not keep page content.
4. The Terms of Service page, if robots.txt allows it. The text is hashed to detect changes; the public Index quotes short excerpts with a link to the source.

That is six files, or at most about eight requests per site per day once redirects and a www fallback are counted.

What it never does

• It never logs in, creates accounts, or touches carts, checkout, or account pages.
• It never solves or bypasses a CAPTCHA or bot challenge. If it is challenged or refused, the Index records that and moves on.
• It never runs a headless browser or executes JavaScript, and never sends a browser-like user agent.
• It never rotates IP addresses or uses proxies.
• It retries at most once, and only after a network error or a 5xx response.

How to stop it

Add this to your robots.txt. Our bot reads it on its next daily run and stops fetching anything else:

User-agent: WalledWebBotDisallow: /

The Index will then show your homepage as "Not tested" because robots.txt bars our bot. It still reads robots.txt itself, which RFC 9309 treats as always permitted, so the public record of your AI rules stays current. To be removed from checking entirely, use the form on this page. Requests are honored within a week, and the Index lists the domain as "Owner requested no checking".