The Walled Web Newsletter

AI bots we track / AI training

Who blocks GPTBot?

GPTBot is OpenAI's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. OpenAI says it obeys robots.txt: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Source

32of 159
sites block it from the whole site
26of 32
of those name it in robots.txt
0of 159
block it from part of the site
10of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking GPTBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

GPTBot 20%
0%10%20%30%2023202420252026GPTBot 20%0%10%20%30%'23'24'25'26
Table view
MonthGPTBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 202320%143
Oct 202323%143
Nov 202325%143
Dec 202326%143
Jan 202426%143
Feb 202426%143
Mar 202424%143
Apr 202424%143
May 202423%143
Jun 202422%143
Jul 202420%143
Aug 202420%143
Sep 202419%143
Oct 202419%143
Nov 202419%143
Dec 202419%143
Jan 202517%143
Feb 202517%143
Mar 202517%143
Apr 202517%143
May 202518%143
Jun 202516%143
Jul 202517%143
Aug 202515%143
Sep 202516%143
Oct 202517%143
Nov 202517%143
Dec 202517%143
Jan 202617%143
Feb 202617%143
Mar 202617%143
Apr 202618%143
May 202618%143
Jun 202619%143
Jul 202619%143
Aug 202618%143
Sep 202618%143
Oct 202620%143

Blocked by name (26)

AmazoneBayAliExpressAirbnbTripadvisorYelpNextdoorGlassdoorThe New York TimesBBCCNNAssociated PressBloombergNPRUSA TodayPoliticoThe EconomistLos Angeles TimesMediumFacebookInstagramLinkedInTikTokQuoraThreadsWebMD

Blocked by the general rules for all crawlers (6)

InstacartReutersRedditXPinterestIMDb

Blocked from part of the site (0)

None.

Named and allowed (10)

Best BuyTemuIndeedThe Wall Street JournalThe AtlanticThe VergeStack OverflowGitHubSupabaseCVS Health

Recent changes

  • Oct 2026
    Airbnb Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
    More blocking
  • Oct 2026
    AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
    More blocking
  • Oct 2026
    Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    More blocking
  • Aug 2026
    Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Less blocking
  • Apr to Jun 2026
    Reuters Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.
    More blocking
  • Apr 2026
    eBay Now blocked: GPTBot, Applebot-Extended and Meta-ExternalAgent.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Oct 2025
    Tripadvisor Now blocked: GPTBot.
    More blocking
  • Sep 2025
    IKEA Now allowed: GPTBot, Google-Extended and CCBot.
    Less blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking