Which websites block AI?

How we check

1. Websites post rules for bots

   Almost every website has a public file called robots.txt that tells bots where they may and may not go. Anyone can read it. Here is the part of The New York Times' file that tells OpenAI's bot to stay out:

   User-agent: GPTBot
   Disallow: /

2. We read those rules every day

   We read them for 170 big sites, like Amazon, Reddit and The New York Times. Old copies saved by the Internet Archive let us go back to Jan 2023.

3. Then we knock on the front door

   robots.txt is a polite request, not a lock. 5 of the 7 AI assistants we track come from companies that say they may ignore robots.txt when a person asks for a page (OpenAI, Perplexity, Google, Meta, Amazon). So we also visit each homepage with a bot that says who it is, and write down whether the site lets it in, turns it away, or asks it to prove it is human.

Three kinds of AI bots

Sites often treat these differently, so the table below shows each one.

All 170 sites

Overall is our one-word summary. Training, Search and Assistants show what the site's rules say to each kind of AI bot: blocked some blocked allowed. Let our bot in? is what happened when we visited the homepage. Click any site for the full story.

170 sites

10 sites let one or two AI companies in and keep the rest out

These sites name one or two AI companies' bots and let them in, while blocking bots from at least three other companies.