Download the data
Everything on this site is free to download, updated after every daily check. The code is Apache-2.0 and the data is CC BY 4.0. Please cite it as "Agent Access Index, The Walled Web" with the date you downloaded it.
| File | Contents |
|---|---|
| index.json | Every site with its current status, lanes, door result and per-agent codes. |
| sites/{domain}.json | Everything on a site page, including the robots.txt text and change events with diffs. |
| changes.json | All AI-relevant change events. |
| trends.json | Monthly shares behind the trend charts. |
| wall.json | Site × agent × month access codes behind the wall. |
| tokens.json | Every user-agent token named in tracked robots.txt files. |
| sites.csv | Flat table of sites and current statuses. |
| changes.csv | Flat table of change events. |
| agents.csv | The agent list with operators, purposes and sources. |
| agent-access-index.sqlite | All of the above as one SQLite database (Datasette-friendly). |
| DATA_DICTIONARY.md | What every field means. |
Generated 20 Oct 2026, 02:23 UTC from the check of 2026-10-19. Raw robots.txt files are kept in the repository's content-addressed blob store, so every derived number can be recomputed. Terms of Service text and llms.txt files are kept privately to detect changes and are not republished.