The Walled Web Newsletter

AI bots we track / AI assistants

Who blocks Meta-ExternalFetcher?

Meta-ExternalFetcher is Meta's AI assistant bot. It visits a page the moment a person asks an AI to. Meta says it may not obey robots.txt: "This crawler may bypass robots.txt rules." Source

25of 159
sites block it from the whole site
15of 25
of those name it in robots.txt
1of 159
block it from part of the site
1of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Meta-ExternalFetcher, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Meta-ExternalFetcher 16%
0%5%10%15%20%2023202420252026Meta-ExternalFetcher 16%0%5%10%15%20%'23'24'25'26
Table view
MonthMeta-ExternalFetcherSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20244%143
Sep 20243%143
Oct 20244%143
Nov 20245%143
Dec 20246%143
Jan 20256%143
Feb 20256%143
Mar 20256%143
Apr 20258%143
May 202510%143
Jun 202510%143
Jul 202510%143
Aug 202510%143
Sep 202510%143
Oct 202512%143
Nov 202513%143
Dec 202513%143
Jan 202612%143
Feb 202613%143
Mar 202613%143
Apr 202613%143
May 202613%143
Jun 202615%143
Jul 202615%143
Aug 202615%143
Sep 202615%143
Oct 202616%143

Blocked by name (15)

AmazonYelpRealtor.comThe New York TimesThe Washington PostBloombergFinancial TimesThe AtlanticThe New YorkerUSA TodayPoliticoThe VergeWIREDXLinkedIn

Blocked by the general rules for all crawlers (10)

InstacartNextdoorThe Wall Street JournalReutersRedditFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Quora

Named and allowed (1)

Indeed

Recent changes

  • Oct 2026
    Pinterest Now blocked: Meta-ExternalFetcher.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Jun 2026
    Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    More blocking
  • Jun 2026
    The Verge Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
    More blocking
  • Apr 2026
    The Atlantic Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added Content-Signal: search=yes, ai-input=no, ai-train=no.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025 to Jan 2026
    CNN Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.
    Mixed
  • Dec 2025
    Amazon Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.
    More blocking
  • Oct 2025
    The New Yorker Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.
    Mixed
  • Oct 2025
    WIRED Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.
    Mixed
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking