The Walled Web Newsletter

AI bots we track / AI training

Who blocks Google-Extended?

Google-Extended is Google's bot that collects pages to train AI (a name sites use to opt out; no bot actually visits under it). Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. Google says it obeys robots.txt: A control token with no crawler of its own. Covers Gemini training and grounding. "Google-Extended does not impact a site's inclusion in Google Search." It does not control AI Overviews or AI Mode. Source

31of 159
sites block it from the whole site
24of 31
of those name it in robots.txt
0of 159
block it from part of the site
8of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Google-Extended, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Google-Extended 20%
0%10%20%30%2023202420252026Google-Extended 20%0%10%20%30%'23'24'25'26
Table view
MonthGoogle-ExtendedSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20235%143
Nov 202313%143
Dec 202314%143
Jan 202415%143
Feb 202415%143
Mar 202416%143
Apr 202417%143
May 202417%143
Jun 202416%143
Jul 202417%143
Aug 202418%143
Sep 202417%143
Oct 202417%143
Nov 202418%143
Dec 202418%143
Jan 202517%143
Feb 202517%143
Mar 202518%143
Apr 202518%143
May 202520%143
Jun 202519%143
Jul 202521%143
Aug 202520%143
Sep 202520%143
Oct 202521%143
Nov 202521%143
Dec 202522%143
Jan 202620%143
Feb 202621%143
Mar 202620%143
Apr 202620%143
May 202620%143
Jun 202620%143
Jul 202620%143
Aug 202620%143
Sep 202620%143
Oct 202620%143

Blocked by name (24)

AmazonAliExpressTripadvisorYelpGlassdoorThe New York TimesBBCBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayPoliticoThe VergeWIREDThe EconomistXFacebookInstagramLinkedInTikTokTumblrThreads

Blocked by the general rules for all crawlers (7)

InstacartNextdoorThe Wall Street JournalReutersRedditPinterestIMDb

Blocked from part of the site (0)

None.

Named and allowed (8)

Best BuyTemuIndeedAxiosQuoraStack OverflowSupabaseCVS Health

Recent changes

  • Oct 2026
    AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
    More blocking
  • Oct 2026
    CNN Now allowed: Google-Extended.
    Less blocking
  • Oct 2026
    Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    More blocking
  • Aug 2026
    Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Less blocking
  • Dec 2025 to Jul 2026
    Craigslist Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.
    Less blocking
  • Mar 2026
    Axios Now allowed: ClaudeBot and Google-Extended.
    Less blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Jun to Sep 2025
    Amazon Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.
    More blocking
  • Sep 2025
    IKEA Now allowed: GPTBot, Google-Extended and CCBot.
    Less blocking
  • Sep 2025
    Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking