The Walled Web Newsletter

AI bots we track / AI training

Who blocks AI2Bot?

AI2Bot is Allen Institute for AI's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Allen Institute for AI makes no robots.txt statement: "This web content is used to train open language models." No robots.txt statement. Source

21of 159
sites block it from the whole site
8of 21
of those name it in robots.txt
1of 159
block it from part of the site
1of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking AI2Bot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

AI2Bot 13%
0%5%10%15%20%2023202420252026AI2Bot 13%0%5%10%15%20%'23'24'25'26
Table view
MonthAI2BotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20244%143
Sep 20243%143
Oct 20243%143
Nov 20243%143
Dec 20244%143
Jan 20253%143
Feb 20253%143
Mar 20254%143
Apr 20254%143
May 20256%143
Jun 20256%143
Jul 20256%143
Aug 20256%143
Sep 20257%143
Oct 20258%143
Nov 20258%143
Dec 20258%143
Jan 20268%143
Feb 20268%143
Mar 20269%143
Apr 202610%143
May 202610%143
Jun 202612%143
Jul 202612%143
Aug 202613%143
Sep 202613%143
Oct 202613%143

Blocked by name (8)

AmazonAirbnbZillowCNNUSA TodayPoliticoThe VergeTikTok

Blocked by the general rules for all crawlers (13)

InstacartYelpNextdoorThe Wall Street JournalReutersRedditXFacebookInstagramLinkedInPinterestThreadsIMDb

Blocked from part of the site (1)

Quora

Named and allowed (1)

Indeed

Recent changes

  • Oct 2026
    Airbnb Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Jun 2026
    The Verge Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
    More blocking
  • Jun 2026
    The Wall Street Journal Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Feb to May 2026
    Pinterest Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Apr 2026
    Zillow Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot and Amazonbot.
    More blocking
  • Mar 2026
    Reuters Now blocked: AI2Bot and MistralAI-User. Now allowed: Claude-User.
    Mixed
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Jun to Sep 2025
    Amazon Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking