For developers
Does my bot get in?
If you build an AI agent or crawler, type the name it uses in robots.txt. We test it against the rules of all 159 sites we could read, using the same RFC 9309 parser as the rest of this site. It runs in your browser; nothing is sent anywhere.
What the result means, and what it doesn't
- Named means the file has a group for your exact name: it addresses your bot directly.
- Via the general rules means your bot falls under
User-agent: *. Many sites block every unnamed crawler this way; it is not about you. - robots.txt is only half the story. The front door result shows what happened when our own bot, which says who it is, asked for the homepage. If a site refused us, it may refuse you, whatever robots.txt says.
- Several AI companies say their assistants may not follow robots.txt at all. Whether yours should is your call; we record what sites ask for.
Free JSON, no key
Everything on this site is published as static JSON with open CORS, rebuilt after every daily run. Code is Apache-2.0; data is CC BY 4.0.
/data/index.json: every site with status, lanes, door result and per-agent access codes./data/sites/{domain}.json: one site in full, including robots.txt text, per-agent access, history and diffs./data/robots-bundle.json: the current robots.txt of every tracked site in one file (what this tool uses)./data/changes.jsonand/changes.xml(RSS): every AI-relevant change since 2023./data/agent-access-index.sqlite: all of it as one SQLite database.
Data as of 19 Oct 2026, 13:46 UTC.