The Walled Web Newsletter

AI bots we track / AI training

Who blocks Meta-ExternalAgent?

Meta-ExternalAgent is Meta's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Meta implies it obeys robots.txt: Crawls "for use cases such as training foundation AI models". Meta says to block it in robots.txt; it has no explicit compliance sentence. Source

33of 159
sites block it from the whole site
24of 33
of those name it in robots.txt
1of 159
block it from part of the site
4of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Meta-ExternalAgent, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Meta-ExternalAgent 21%
0%10%20%30%2023202420252026Meta-ExternalAgent 21%0%10%20%30%'23'24'25'26
Table view
MonthMeta-ExternalAgentSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20245%143
Sep 20246%143
Oct 20247%143
Nov 202410%143
Dec 202412%143
Jan 202511%143
Feb 202511%143
Mar 202513%143
Apr 202513%143
May 202514%143
Jun 202514%143
Jul 202514%143
Aug 202514%143
Sep 202517%143
Oct 202517%143
Nov 202517%143
Dec 202517%143
Jan 202617%143
Feb 202618%143
Mar 202619%143
Apr 202620%143
May 202620%143
Jun 202621%143
Jul 202620%143
Aug 202621%143
Sep 202621%143
Oct 202621%143

Blocked by name (24)

AmazoneBayInstacartTripadvisorYelpZillowRealtor.comThe New York TimesThe Washington PostThe GuardianBBCBloombergFinancial TimesThe AtlanticThe New YorkerUSA TodayPoliticoThe VergeWIREDMediumXLinkedInTikTokTumblr

Blocked by the general rules for all crawlers (9)

NextdoorThe Wall Street JournalReutersRedditFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Quora

Named and allowed (4)

NordstromAirbnbIndeedGOV.UK

Recent changes

  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Dec 2025 to Jul 2026
    Craigslist Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.
    Less blocking
  • Jun 2026
    The Verge Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
    More blocking
  • Apr 2026
    eBay Now blocked: GPTBot, Applebot-Extended and Meta-ExternalAgent.
    More blocking
  • Apr 2026
    The Atlantic Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added Content-Signal: search=yes, ai-input=no, ai-train=no.
    More blocking
  • Apr 2026
    Zillow Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot and Amazonbot.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Feb 2026
    Tripadvisor Now blocked: CCBot, Meta-ExternalAgent and Amazonbot.
    More blocking
  • Oct 2025 to Jan 2026
    CNN Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.
    Mixed
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Jun to Sep 2025
    Amazon Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.
    More blocking
  • Sep 2025
    Medium Now blocked: Meta-ExternalAgent and Amazonbot.
    More blocking