The Walled Web Newsletter

AI bots we track / AI training

Who blocks CCBot?

CCBot is Common Crawl's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Common Crawl says it obeys robots.txt: Builds an open web archive that is widely used downstream to train language models. Its FAQ says it checks robots.txt first. Source

39of 159
sites block it from the whole site
28of 39
of those name it in robots.txt
1of 159
block it from part of the site
3of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking CCBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

CCBot 25%
0%10%20%30%2023202420252026CCBot 25%0%10%20%30%'23'24'25'26
Table view
MonthCCBotSites with a file
Jan 20234%143
Feb 20235%143
Mar 20235%143
Apr 20235%143
May 20236%143
Jun 20236%143
Jul 20235%143
Aug 20236%143
Sep 20239%143
Oct 202313%143
Nov 202314%143
Dec 202315%143
Jan 202416%143
Feb 202416%143
Mar 202417%143
Apr 202419%143
May 202420%143
Jun 202419%143
Jul 202420%143
Aug 202421%143
Sep 202420%143
Oct 202421%143
Nov 202421%143
Dec 202422%143
Jan 202521%143
Feb 202521%143
Mar 202522%143
Apr 202522%143
May 202522%143
Jun 202522%143
Jul 202523%143
Aug 202522%143
Sep 202522%143
Oct 202522%143
Nov 202522%143
Dec 202523%143
Jan 202623%143
Feb 202624%143
Mar 202624%143
Apr 202624%143
May 202624%143
Jun 202624%143
Jul 202624%143
Aug 202624%143
Sep 202624%143
Oct 202625%143

Blocked by name (28)

AmazoneBayAliExpressTripadvisorYelpEventbriteGlassdoorThe New York TimesThe Washington PostThe GuardianBBCCNNAssociated PressBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayAxiosPoliticoThe VergeWIREDThe EconomistLinkedInTikTokTumblrWebMD

Blocked by the general rules for all crawlers (11)

InstacartNextdoorThe Wall Street JournalReutersRedditXFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Quora

Named and allowed (3)

Best BuyIndeedCVS Health

Recent changes

  • Oct 2026
    AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Feb 2026
    Tripadvisor Now blocked: CCBot, Meta-ExternalAgent and Amazonbot.
    More blocking
  • Jan 2026
    eBay Now blocked: ClaudeBot, CCBot, anthropic-ai, PerplexityBot and Amazonbot.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Sep 2025
    Craigslist Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.
    Less blocking
  • Sep 2025
    IKEA Now allowed: GPTBot, Google-Extended and CCBot.
    Less blocking
  • Sep 2025
    Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Aug 2025
    OpenTable Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 10 more.
    More blocking
  • Jul 2025
    IMDb Now blocked: GPTBot, Google-Extended, CCBot and anthropic-ai.
    More blocking
  • May 2025
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking