The Walled Web Newsletter

AI bots we track / AI training

Who blocks ClaudeBot?

ClaudeBot is Anthropic's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Anthropic says it obeys robots.txt: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Source

41of 159
sites block it from the whole site
33of 41
of those name it in robots.txt
0of 159
block it from part of the site
8of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking ClaudeBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

ClaudeBot 26%
0%10%20%30%2023202420252026ClaudeBot 26%0%10%20%30%'23'24'25'26
Table view
MonthClaudeBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20246%143
Apr 20248%143
May 202411%143
Jun 202413%143
Jul 202414%143
Aug 202415%143
Sep 202416%143
Oct 202417%143
Nov 202417%143
Dec 202418%143
Jan 202517%143
Feb 202517%143
Mar 202518%143
Apr 202518%143
May 202520%143
Jun 202520%143
Jul 202520%143
Aug 202520%143
Sep 202519%143
Oct 202521%143
Nov 202521%143
Dec 202522%143
Jan 202622%143
Feb 202622%143
Mar 202622%143
Apr 202622%143
May 202622%143
Jun 202622%143
Jul 202622%143
Aug 202623%143
Sep 202623%143
Oct 202626%143

Blocked by name (33)

AmazoneBayAliExpressAirbnbTripadvisorYelpGlassdoorThe New York TimesThe Washington PostThe GuardianBBCCNNAssociated PressBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayPoliticoThe VergeWIREDThe EconomistLos Angeles TimesMediumFacebookInstagramLinkedInTikTokQuoraTumblrThreadsWebMD

Blocked by the general rules for all crawlers (8)

InstacartNextdoorThe Wall Street JournalReutersRedditXPinterestIMDb

Blocked from part of the site (0)

None.

Named and allowed (8)

Best BuyMacy'sTemuIndeedGitHubSupabaseHumanaCVS Health

Recent changes

  • Oct 2026
    Airbnb Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
    More blocking
  • Oct 2026
    AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
    More blocking
  • Oct 2026
    Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    More blocking
  • May to Oct 2026
    Glassdoor Now blocked: ClaudeBot.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Aug 2026
    The Economist Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.
    More blocking
  • Aug 2026
    Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Less blocking
  • Jun 2026
    Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    More blocking
  • Mar 2026
    Axios Now allowed: ClaudeBot and Google-Extended.
    Less blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Jan 2026
    eBay Now blocked: ClaudeBot, CCBot, anthropic-ai, PerplexityBot and Amazonbot.
    More blocking
  • Oct 2025
    The New Yorker Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.
    Mixed