The Walled Web Newsletter

AI bots we track / AI search

Who blocks Amzn-SearchBot?

Amzn-SearchBot is Amazon's bot for AI search. Blocking it usually means the site stops showing up as a source in that company's AI answers. Amazon says it obeys robots.txt: "Amzn-SearchBot does not crawl content for generative AI model training." Source

16of 159
sites block it from the whole site
4of 16
of those name it in robots.txt
1of 159
block it from part of the site
2of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Amzn-SearchBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Amzn-SearchBot 9%
0%5%10%15%20%2023202420252026Amzn-SearchBot 9%0%5%10%15%20%'23'24'25'26
Table view
MonthAmzn-SearchBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20244%143
Sep 20243%143
Oct 20243%143
Nov 20243%143
Dec 20244%143
Jan 20253%143
Feb 20253%143
Mar 20254%143
Apr 20254%143
May 20255%143
Jun 20255%143
Jul 20255%143
Aug 20254%143
Sep 20254%143
Oct 20254%143
Nov 20254%143
Dec 20255%143
Jan 20266%143
Feb 20266%143
Mar 20266%143
Apr 20267%143
May 20267%143
Jun 20269%143
Jul 20269%143
Aug 20269%143
Sep 20269%143
Oct 20269%143

Blocked by name (4)

The GuardianBBCCNNUSA Today

Blocked by the general rules for all crawlers (12)

InstacartYelpNextdoorThe Wall Street JournalReutersRedditXFacebookInstagramLinkedInPinterestThreads

Blocked from part of the site (1)

Quora

Named and allowed (2)

TwitchIMDb

Recent changes

  • Apr to Jun 2026
    Reuters Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.
    More blocking
  • Apr to Jun 2026
    USA Today Now blocked: Amzn-SearchBot, Meta-WebIndexer, Perplexity-User, Google-Agent and Amzn-User.
    More blocking
  • Jun 2026
    The Wall Street Journal Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Feb to May 2026
    Pinterest Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Jan 2026
    BBC Now blocked: Amzn-SearchBot and Amzn-User.
    More blocking
  • Oct 2025 to Jan 2026
    CNN Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.
    Mixed
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Aug 2025
    OpenTable Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 10 more.
    More blocking
  • May 2025
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Sep 2024 to May 2025
    Threads Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Mar 2025
    Instacart Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 14 more.
    More blocking