The Walled Web Newsletter

AI bots we track / AI search

Who blocks Claude-SearchBot?

Claude-SearchBot is Anthropic's bot for AI search. Blocking it usually means the site stops showing up as a source in that company's AI answers. Anthropic says it obeys robots.txt: "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization." Source

27of 159
sites block it from the whole site
16of 27
of those name it in robots.txt
1of 159
block it from part of the site
7of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Claude-SearchBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Claude-SearchBot 16%
0%5%10%15%20%2023202420252026Claude-SearchBot 16%0%5%10%15%20%'23'24'25'26
Table view
MonthClaude-SearchBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20244%143
Sep 20243%143
Oct 20243%143
Nov 20243%143
Dec 20244%143
Jan 20253%143
Feb 20253%143
Mar 20254%143
Apr 20254%143
May 20256%143
Jun 20256%143
Jul 20256%143
Aug 20257%143
Sep 20258%143
Oct 202510%143
Nov 202510%143
Dec 202511%143
Jan 202610%143
Feb 202611%143
Mar 202611%143
Apr 202613%143
May 202613%143
Jun 202615%143
Jul 202615%143
Aug 202616%143
Sep 202616%143
Oct 202616%143

Blocked by name (16)

AmazonYelpThe New York TimesThe GuardianCNNAssociated PressThe AtlanticThe New YorkerNPRUSA TodayPoliticoThe VergeWIREDThe EconomistTikTokQuora

Blocked by the general rules for all crawlers (11)

InstacartNextdoorThe Wall Street JournalReutersRedditXFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Bloomberg

Named and allowed (7)

eBayTemuExpediaVrboIndeedLinkedInHumana

Recent changes

  • Oct 2026
    Bloomberg Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.
    Less blocking
  • Oct 2026
    NPR Now blocked: Claude-SearchBot and Claude-User.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Aug 2026
    The Economist Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.
    More blocking
  • Mar to Jul 2026
    LinkedIn Now blocked: Bytespider. Now allowed: Claude-SearchBot.
    Mixed
  • Jun 2026
    Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    More blocking
  • Mar to Jun 2026
    CNN Now blocked: Claude-SearchBot, Claude-User, Perplexity-User and MistralAI-User.
    More blocking
  • Jun 2026
    The Verge Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
    More blocking
  • Jun 2026
    The Wall Street Journal Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Feb to May 2026
    Pinterest Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Apr 2026
    The Atlantic Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added Content-Signal: search=yes, ai-input=no, ai-train=no.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking