The Walled Web Newsletter

AI bots we track / AI training

Who blocks Applebot-Extended?

Applebot-Extended is Apple's bot that collects pages to train AI (a name sites use to opt out; no bot actually visits under it). Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Apple says it obeys robots.txt: "Applebot-Extended does not crawl webpages." It only decides whether pages Applebot crawled may train Apple's models. Source

38of 159
sites block it from the whole site
29of 38
of those name it in robots.txt
0of 159
block it from part of the site
1of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Applebot-Extended, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Applebot-Extended 24%
0%10%20%30%2023202420252026Applebot-Extended 24%0%10%20%30%'23'24'25'26
Table view
MonthApplebot-ExtendedSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20249%143
Aug 202411%143
Sep 202413%143
Oct 202413%143
Nov 202415%143
Dec 202416%143
Jan 202515%143
Feb 202515%143
Mar 202516%143
Apr 202517%143
May 202519%143
Jun 202519%143
Jul 202519%143
Aug 202518%143
Sep 202519%143
Oct 202520%143
Nov 202520%143
Dec 202520%143
Jan 202620%143
Feb 202620%143
Mar 202620%143
Apr 202622%143
May 202622%143
Jun 202622%143
Jul 202622%143
Aug 202622%143
Sep 202622%143
Oct 202624%143

Blocked by name (29)

eBayAirbnbTripadvisorZillowGlassdoorThe New York TimesThe Washington PostThe GuardianBBCCNNReutersAssociated PressBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayPoliticoThe VergeWIREDThe EconomistMediumFacebookInstagramTikTokQuoraTumblrThreads

Blocked by the general rules for all crawlers (9)

InstacartYelpNextdoorThe Wall Street JournalRedditXLinkedInPinterestIMDb

Blocked from part of the site (0)

None.

Named and allowed (1)

Indeed

Recent changes

  • Oct 2026
    Airbnb Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
    More blocking
  • Oct 2026
    Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Aug 2026
    Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Less blocking
  • Jun 2026
    Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    More blocking
  • Apr 2026
    eBay Now blocked: GPTBot, Applebot-Extended and Meta-ExternalAgent.
    More blocking
  • Apr 2026
    Zillow Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot and Amazonbot.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Sep 2025
    Craigslist Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.
    Less blocking
  • Sep 2025
    Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking