Is AI getting blocked more over time?
In Jan 2023, 4% of these sites told OpenAI's GPTBot to stay out. Now it is 20%. Each line follows the same 143 sites every month, using old copies from the Internet Archive and our own daily check.
The 4% at the start is from before GPTBot existed: those sites already kept out every bot they don't name. We follow the same sites every month so the lines can't rise just because more sites were added. The other 27 sites are on the home page but not in these charts. How this works
AI training bots
Share of sites that tell each bot to stay out of the whole site. GPTBot is OpenAI's, ClaudeBot is Anthropic's, Google-Extended is Google's AI training opt-out, and CCBot belongs to Common Crawl, whose data many AI companies use. Other training bots are in gray.
GPTBotClaudeBotGoogle-ExtendedCCBotOthers
Table view
| Month | GPTBot | ClaudeBot | Google-Extended | CCBot | Sites with a file |
|---|---|---|---|---|---|
| Jan 2023 | 4% | 4% | 4% | 4% | 143 |
| Feb 2023 | 4% | 4% | 4% | 5% | 143 |
| Mar 2023 | 4% | 4% | 4% | 5% | 143 |
| Apr 2023 | 4% | 4% | 4% | 5% | 143 |
| May 2023 | 4% | 4% | 4% | 6% | 143 |
| Jun 2023 | 4% | 4% | 4% | 6% | 143 |
| Jul 2023 | 4% | 4% | 4% | 5% | 143 |
| Aug 2023 | 4% | 4% | 4% | 6% | 143 |
| Sep 2023 | 20% | 4% | 4% | 9% | 143 |
| Oct 2023 | 23% | 4% | 5% | 13% | 143 |
| Nov 2023 | 25% | 4% | 13% | 14% | 143 |
| Dec 2023 | 26% | 4% | 14% | 15% | 143 |
| Jan 2024 | 26% | 4% | 15% | 16% | 143 |
| Feb 2024 | 26% | 4% | 15% | 16% | 143 |
| Mar 2024 | 24% | 6% | 16% | 17% | 143 |
| Apr 2024 | 24% | 8% | 17% | 19% | 143 |
| May 2024 | 23% | 11% | 17% | 20% | 143 |
| Jun 2024 | 22% | 13% | 16% | 19% | 143 |
| Jul 2024 | 20% | 14% | 17% | 20% | 143 |
| Aug 2024 | 20% | 15% | 18% | 21% | 143 |
| Sep 2024 | 19% | 16% | 17% | 20% | 143 |
| Oct 2024 | 19% | 17% | 17% | 21% | 143 |
| Nov 2024 | 19% | 17% | 18% | 21% | 143 |
| Dec 2024 | 19% | 18% | 18% | 22% | 143 |
| Jan 2025 | 17% | 17% | 17% | 21% | 143 |
| Feb 2025 | 17% | 17% | 17% | 21% | 143 |
| Mar 2025 | 17% | 18% | 18% | 22% | 143 |
| Apr 2025 | 17% | 18% | 18% | 22% | 143 |
| May 2025 | 18% | 20% | 20% | 22% | 143 |
| Jun 2025 | 16% | 20% | 19% | 22% | 143 |
| Jul 2025 | 17% | 20% | 21% | 23% | 143 |
| Aug 2025 | 15% | 20% | 20% | 22% | 143 |
| Sep 2025 | 16% | 19% | 20% | 22% | 143 |
| Oct 2025 | 17% | 21% | 21% | 22% | 143 |
| Nov 2025 | 17% | 21% | 21% | 22% | 143 |
| Dec 2025 | 17% | 22% | 22% | 23% | 143 |
| Jan 2026 | 17% | 22% | 20% | 23% | 143 |
| Feb 2026 | 17% | 22% | 21% | 24% | 143 |
| Mar 2026 | 17% | 22% | 20% | 24% | 143 |
| Apr 2026 | 18% | 22% | 20% | 24% | 143 |
| May 2026 | 18% | 22% | 20% | 24% | 143 |
| Jun 2026 | 19% | 22% | 20% | 24% | 143 |
| Jul 2026 | 19% | 22% | 20% | 24% | 143 |
| Aug 2026 | 18% | 23% | 20% | 24% | 143 |
| Sep 2026 | 18% | 23% | 20% | 24% | 143 |
| Oct 2026 | 20% | 26% | 20% | 25% | 143 |
AI search and AI assistants
These visit pages when people ask AI tools for something. ChatGPT-User, Claude-User and Perplexity-User fetch pages for a person in real time; OAI-SearchBot powers ChatGPT search. Several of these companies say real-time visits may ignore robots.txt.
ChatGPT-UserClaude-UserPerplexity-UserOAI-SearchBotOthers
Table view
| Month | ChatGPT-User | Claude-User | Perplexity-User | OAI-SearchBot | Sites with a file |
|---|---|---|---|---|---|
| Jan 2023 | 4% | 4% | 4% | 4% | 143 |
| Feb 2023 | 4% | 4% | 4% | 4% | 143 |
| Mar 2023 | 4% | 4% | 4% | 4% | 143 |
| Apr 2023 | 4% | 4% | 4% | 4% | 143 |
| May 2023 | 4% | 4% | 4% | 4% | 143 |
| Jun 2023 | 4% | 4% | 4% | 4% | 143 |
| Jul 2023 | 4% | 4% | 4% | 4% | 143 |
| Aug 2023 | 4% | 4% | 4% | 4% | 143 |
| Sep 2023 | 7% | 4% | 4% | 4% | 143 |
| Oct 2023 | 8% | 4% | 4% | 4% | 143 |
| Nov 2023 | 9% | 4% | 4% | 4% | 143 |
| Dec 2023 | 9% | 4% | 4% | 4% | 143 |
| Jan 2024 | 10% | 4% | 4% | 4% | 143 |
| Feb 2024 | 10% | 4% | 4% | 4% | 143 |
| Mar 2024 | 11% | 4% | 4% | 4% | 143 |
| Apr 2024 | 13% | 5% | 5% | 5% | 143 |
| May 2024 | 13% | 4% | 4% | 4% | 143 |
| Jun 2024 | 12% | 4% | 4% | 4% | 143 |
| Jul 2024 | 12% | 4% | 4% | 4% | 143 |
| Aug 2024 | 13% | 4% | 4% | 6% | 143 |
| Sep 2024 | 10% | 3% | 3% | 5% | 143 |
| Oct 2024 | 10% | 3% | 3% | 5% | 143 |
| Nov 2024 | 10% | 3% | 3% | 6% | 143 |
| Dec 2024 | 11% | 4% | 4% | 6% | 143 |
| Jan 2025 | 10% | 3% | 3% | 6% | 143 |
| Feb 2025 | 10% | 3% | 3% | 6% | 143 |
| Mar 2025 | 11% | 4% | 4% | 6% | 143 |
| Apr 2025 | 10% | 4% | 4% | 6% | 143 |
| May 2025 | 12% | 6% | 6% | 8% | 143 |
| Jun 2025 | 10% | 6% | 6% | 6% | 143 |
| Jul 2025 | 11% | 6% | 7% | 8% | 143 |
| Aug 2025 | 10% | 7% | 6% | 7% | 143 |
| Sep 2025 | 12% | 8% | 8% | 8% | 143 |
| Oct 2025 | 12% | 10% | 10% | 8% | 143 |
| Nov 2025 | 12% | 10% | 10% | 8% | 143 |
| Dec 2025 | 13% | 11% | 12% | 9% | 143 |
| Jan 2026 | 13% | 10% | 11% | 8% | 143 |
| Feb 2026 | 13% | 11% | 12% | 9% | 143 |
| Mar 2026 | 13% | 10% | 12% | 10% | 143 |
| Apr 2026 | 13% | 11% | 13% | 10% | 143 |
| May 2026 | 13% | 13% | 14% | 10% | 143 |
| Jun 2026 | 13% | 15% | 16% | 10% | 143 |
| Jul 2026 | 13% | 15% | 16% | 10% | 143 |
| Aug 2026 | 14% | 17% | 17% | 12% | 143 |
| Sep 2026 | 13% | 17% | 17% | 12% | 143 |
| Oct 2026 | 13% | 17% | 17% | 11% | 143 |
By kind of AI bot
Share of sites that block at least one bot of each kind.
AI training botsAI search botsAI assistants
Table view
| Month | AI training bots | AI search bots | AI assistants | Sites with a file |
|---|---|---|---|---|
| Jan 2023 | 6% | 4% | 4% | 143 |
| Feb 2023 | 6% | 4% | 4% | 143 |
| Mar 2023 | 6% | 4% | 4% | 143 |
| Apr 2023 | 6% | 4% | 4% | 143 |
| May 2023 | 8% | 4% | 4% | 143 |
| Jun 2023 | 7% | 4% | 4% | 143 |
| Jul 2023 | 6% | 4% | 4% | 143 |
| Aug 2023 | 7% | 4% | 4% | 143 |
| Sep 2023 | 21% | 4% | 7% | 143 |
| Oct 2023 | 26% | 4% | 8% | 143 |
| Nov 2023 | 29% | 5% | 9% | 143 |
| Dec 2023 | 29% | 6% | 9% | 143 |
| Jan 2024 | 30% | 7% | 10% | 143 |
| Feb 2024 | 30% | 6% | 10% | 143 |
| Mar 2024 | 29% | 9% | 11% | 143 |
| Apr 2024 | 29% | 10% | 13% | 143 |
| May 2024 | 30% | 13% | 13% | 143 |
| Jun 2024 | 30% | 13% | 12% | 143 |
| Jul 2024 | 29% | 15% | 12% | 143 |
| Aug 2024 | 29% | 17% | 13% | 143 |
| Sep 2024 | 29% | 16% | 10% | 143 |
| Oct 2024 | 29% | 17% | 11% | 143 |
| Nov 2024 | 29% | 17% | 13% | 143 |
| Dec 2024 | 29% | 18% | 13% | 143 |
| Jan 2025 | 29% | 17% | 12% | 143 |
| Feb 2025 | 28% | 17% | 13% | 143 |
| Mar 2025 | 29% | 19% | 14% | 143 |
| Apr 2025 | 29% | 19% | 14% | 143 |
| May 2025 | 29% | 20% | 16% | 143 |
| Jun 2025 | 29% | 18% | 14% | 143 |
| Jul 2025 | 29% | 20% | 15% | 143 |
| Aug 2025 | 27% | 19% | 15% | 143 |
| Sep 2025 | 27% | 22% | 18% | 143 |
| Oct 2025 | 27% | 22% | 20% | 143 |
| Nov 2025 | 27% | 22% | 20% | 143 |
| Dec 2025 | 28% | 23% | 20% | 143 |
| Jan 2026 | 27% | 23% | 20% | 143 |
| Feb 2026 | 28% | 24% | 20% | 143 |
| Mar 2026 | 28% | 24% | 20% | 143 |
| Apr 2026 | 29% | 25% | 20% | 143 |
| May 2026 | 29% | 25% | 20% | 143 |
| Jun 2026 | 29% | 25% | 20% | 143 |
| Jul 2026 | 28% | 24% | 20% | 143 |
| Aug 2026 | 28% | 25% | 21% | 143 |
| Sep 2026 | 28% | 25% | 20% | 143 |
| Oct 2026 | 30% | 26% | 20% | 143 |
Overall
Share of sites by what their rules say to the 12 major AI bots.
Blocks every major AI botBlocks someNo AI-specific blocks
Table view
| Month | Blocks every major AI bot | Blocks some | No AI-specific blocks | Sites with a file |
|---|---|---|---|---|
| Jan 2023 | 4% | 1% | 96% | 143 |
| Feb 2023 | 4% | 1% | 95% | 143 |
| Mar 2023 | 4% | 1% | 95% | 143 |
| Apr 2023 | 4% | 1% | 95% | 143 |
| May 2023 | 4% | 2% | 94% | 143 |
| Jun 2023 | 4% | 2% | 94% | 143 |
| Jul 2023 | 4% | 1% | 95% | 143 |
| Aug 2023 | 4% | 1% | 94% | 143 |
| Sep 2023 | 4% | 16% | 80% | 143 |
| Oct 2023 | 4% | 20% | 76% | 143 |
| Nov 2023 | 4% | 23% | 73% | 143 |
| Dec 2023 | 4% | 24% | 73% | 143 |
| Jan 2024 | 4% | 24% | 72% | 143 |
| Feb 2024 | 4% | 24% | 72% | 143 |
| Mar 2024 | 4% | 23% | 73% | 143 |
| Apr 2024 | 5% | 23% | 72% | 143 |
| May 2024 | 4% | 24% | 71% | 143 |
| Jun 2024 | 4% | 25% | 71% | 143 |
| Jul 2024 | 4% | 24% | 72% | 143 |
| Aug 2024 | 4% | 24% | 73% | 143 |
| Sep 2024 | 3% | 24% | 73% | 143 |
| Oct 2024 | 3% | 24% | 73% | 143 |
| Nov 2024 | 3% | 25% | 72% | 143 |
| Dec 2024 | 4% | 24% | 72% | 143 |
| Jan 2025 | 3% | 24% | 73% | 143 |
| Feb 2025 | 3% | 24% | 73% | 143 |
| Mar 2025 | 3% | 24% | 73% | 143 |
| Apr 2025 | 3% | 24% | 73% | 143 |
| May 2025 | 4% | 24% | 72% | 143 |
| Jun 2025 | 4% | 24% | 72% | 143 |
| Jul 2025 | 4% | 23% | 73% | 143 |
| Aug 2025 | 4% | 22% | 74% | 143 |
| Sep 2025 | 5% | 21% | 74% | 143 |
| Oct 2025 | 6% | 21% | 73% | 143 |
| Nov 2025 | 6% | 21% | 73% | 143 |
| Dec 2025 | 6% | 21% | 73% | 143 |
| Jan 2026 | 5% | 22% | 73% | 143 |
| Feb 2026 | 6% | 22% | 73% | 143 |
| Mar 2026 | 6% | 22% | 73% | 143 |
| Apr 2026 | 6% | 22% | 72% | 143 |
| May 2026 | 7% | 21% | 72% | 143 |
| Jun 2026 | 7% | 21% | 72% | 143 |
| Jul 2026 | 7% | 20% | 73% | 143 |
| Aug 2026 | 7% | 20% | 73% | 143 |
| Sep 2026 | 7% | 20% | 73% | 143 |
| Oct 2026 | 8% | 21% | 71% | 143 |
By type of site
Share of sites of each type that block at least one AI training bot. Same scale in every panel, 0 to 100%.
10 sites let one or two AI companies in and keep the rest out
These sites name one or two AI companies' bots and let them in, while blocking bots from at least three other companies. Where a public agreement between the two companies exists, it is linked. We never assume one.
| Site | Lets in | Keeps out | Since | Public agreement, if sourced |
|---|---|---|---|---|
| Axiosaxios.com | Google Google-Extended | 3 bots from 3 other companies | Mar 2026 | · |
| The New York Timesnytimes.com | Amazon Amazonbot | 16 bots from 9 other companies | Oct 2025 | · |
| The Vergetheverge.com | Amazon Amazonbot | 15 bots from 11 other companies | May 2025 | · |
| The Atlantictheatlantic.com | OpenAI GPTBot, OAI-SearchBot, ChatGPT-User DuckDuckGo DuckAssistBot |
13 bots from 9 other companies | Jul 2024 | Content and product partnership with The Atlantic. The Hollywood Reporter, 29 May 2024 |
| The Wall Street Journalwsj.com | OpenAI GPTBot, OAI-SearchBot, ChatGPT-User | 21 bots from 11 other companies | Jun 2024 | Multi-year content partnership with News Corp (WSJ and others). News Corp, 22 May 2024 |
| Airbnbairbnb.com | Meta Meta-ExternalAgent Perplexity PerplexityBot |
5 bots from 5 other companies | before 2023, or no record | · |
| Tripadvisortripadvisor.com | Perplexity PerplexityBot | 8 bots from 8 other companies | before 2023, or no record | · |
| Pinterestpinterest.com | DuckDuckGo DuckAssistBot | 23 bots from 11 other companies | before 2023, or no record | · |
| Quoraquora.com | Google Google-Extended Amazon Amazonbot |
9 bots from 5 other companies | before 2023, or no record | · |
| IMDbimdb.com | Amazon Amazonbot, Amzn-SearchBot, Amzn-User DuckDuckGo DuckAssistBot |
20 bots from 10 other companies | before 2023, or no record | · |
Sites that signed a deal with OpenAI
Of the ten tracked sites whose owners announced an agreement with OpenAI, nine changed robots.txt to let GPTBot back in within about three months of the announcement, two of them before it was made public. Reddit still blocks GPTBot: its access is through Reddit's data API, not crawling. Dates of robots.txt changes come from the Internet Archive's monthly copies, so each is a window, not a day.
| Site | Agreement announced | GPTBot let back in | Timing | Still blocks (other companies' bots) |
|---|---|---|---|---|
| Financial Timesft.com | 29 Apr 2024 Strategic partnership and licensing agreement with the FT. Financial Times |
1 Jul 2024 to 1 Aug 2024 |
After the announcement | 9 other AI bots |
| Stack Overflowstackoverflow.com | 6 May 2024 API partnership with Stack Overflow (OverflowAPI). Stack Overflow |
1 Mar 2024 to 1 Apr 2024 |
Before the announcement | none |
| Redditreddit.com | 16 May 2024 Partnership giving OpenAI access to Reddit's Data API. Reddit |
Still blocked | 21 other AI bots | |
| The Wall Street Journalwsj.com | 22 May 2024 Multi-year content partnership with News Corp (WSJ and others). News Corp |
1 May 2024 to 1 Jun 2024 |
Around the announcement | 21 other AI bots |
| The Atlantictheatlantic.com | 29 May 2024 Content and product partnership with The Atlantic. The Hollywood Reporter |
1 May 2024 to 1 Jul 2024 |
Around the announcement | 13 other AI bots |
| The Vergetheverge.com | 29 May 2024 Content and product partnership with Vox Media (The Verge and others). The Hollywood Reporter |
1 Jun 2024 to 1 Jul 2024 |
After the announcement | 14 other AI bots |
| The New Yorkernewyorker.com | 20 Aug 2024 Multi-year partnership with Condé Nast (The New Yorker, WIRED and others). InPublishing |
1 May 2024 to 1 Sep 2024 |
Around the announcement | 15 other AI bots |
| WIREDwired.com | 20 Aug 2024 Multi-year partnership with Condé Nast (The New Yorker, WIRED and others). InPublishing |
1 May 2024 to 1 Sep 2024 |
Around the announcement | 15 other AI bots |
| Axiosaxios.com | 15 Jan 2025 Three-year partnership with Axios. Columbia Journalism Review |
1 Jan 2025 to 1 Feb 2025 |
Around the announcement | 3 other AI bots |
| The Washington Postwashingtonpost.com | 22 Apr 2025 Strategic partnership with The Washington Post. Editor & Publisher |
1 Mar 2025 to 1 Apr 2025 |
Before the announcement | 8 other AI bots |
Agreements are listed only with a source a reader can open, checked 19 Oct 2026. We don't claim the deal caused the change. We only put two public dates side by side.
Bot names that aren't on any official list
Sites' rules files also name AI bots beyond the 24 we track. Some point to xAI, the company behind Grok, which has never published them: Grok-DeepSearch (2 sites), GrokBot (2 sites), xAI-Grok (2 sites), Grok (1 site). xAI publishes no bot names at all. A rule for a name no real bot uses blocks nothing, but it shows which names site owners have added.
| Name in robots.txt | Sites | Is it a published name? | Examples |
|---|---|---|---|
| FacebookBot | 18 | Not on our checked list | axios.com, cnn.com, ft.com, glassdoor.com and 14 more |
| PetalBot | 18 | Not on our checked list | amazon.com, bbc.com, bloomberg.com, cnn.com and 14 more |
| Claude-Web | 17 | Not on our checked list | amazon.com, apnews.com, bbc.com, bestbuy.com and 13 more |
| cohere-ai | 17 | Not on our checked list | airbnb.com, amazon.com, apnews.com, bbc.com and 13 more |
| Scrapy | 16 | Not on our checked list | amazon.com, axios.com, bbc.com, cnn.com and 12 more |
| YouBot | 15 | Not on our checked list | amazon.com, bbc.com, bloomberg.com, cnn.com and 11 more |
| omgili | 14 | Not on our checked list | amazon.com, bbc.com, cnn.com, ft.com and 10 more |
| omgilibot | 13 | Not on our checked list | bbc.com, cnn.com, ft.com, glassdoor.com and 9 more |
| ImagesiftBot | 11 | Not on our checked list | airbnb.com, amazon.com, axios.com, cnn.com and 7 more |
| Timpibot | 11 | Not on our checked list | amazon.com, apnews.com, cnn.com, indeed.com and 7 more |
| cohere-training-data-crawler | 8 | Not on our checked list | amazon.com, cnn.com, indeed.com, newyorker.com and 4 more |
| webzio-extended | 8 | Not on our checked list | airbnb.com, amazon.com, cnn.com, indeed.com and 4 more |
| FireCrawlAgent | 7 | Not on our checked list | bbc.com, newyorker.com, nytimes.com, redfin.com and 3 more |
| Gemini-Deep-Research | 6 | Not on our checked list | amazon.com, bestbuy.com, indeed.com, tiktok.com and 2 more |
| Ai2Bot-Dolma | 5 | Not on our checked list | amazon.com, cnn.com, theverge.com, usatoday.com and 1 more |
| AI2Bot-DeepResearchEval | 4 | Not on our checked list | amazon.com, theverge.com, usatoday.com, zillow.com |
| DeepSeekBot | 4 | Not on our checked list | amazon.com, indeed.com, theverge.com, usatoday.com |
| GoogleAgent-Mariner | 3 | Not on our checked list | amazon.com, tiktok.com, usatoday.com |
| meta-externalads | 3 | Not on our checked list | pinterest.com, reuters.com, x.com |
| Webzio | 3 | Not on our checked list | indeed.com, newyorker.com, wired.com |
| Grok-DeepSearch | 2 | No: the company publishes no such name | amazon.com, indeed.com |
| GrokBot | 2 | No: the company publishes no such name | amazon.com, indeed.com |
| Manus-User | 2 | Not on our checked list | amazon.com, usatoday.com |
| TwinAgent | 2 | Not on our checked list | amazon.com, usatoday.com |
| xAI-Grok | 2 | No: the company publishes no such name | amazon.com, indeed.com |
| Amazonbot-Video | 1 | Not on our checked list | twitch.tv |
| Anchor Browser | 1 | Not on our checked list | theverge.com |
| Big Sur AI | 1 | Not on our checked list | usatoday.com |
| bigsur.ai | 1 | Not on our checked list | usatoday.com |
| Browse-AI | 1 | Not on our checked list | usatoday.com |