The Walled Web Newsletter

AI bots we track / AI assistants

Who blocks ChatGPT-User?

ChatGPT-User is OpenAI's AI assistant bot. It visits a page the moment a person asks an AI to. OpenAI says it may not obey robots.txt: "Because these actions are initiated by a user, robots.txt rules may not apply." Source

21of 159
sites block it from the whole site
13of 21
of those name it in robots.txt
1of 159
block it from part of the site
15of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking ChatGPT-User, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

ChatGPT-User 13%
0%5%10%15%20%2023202420252026ChatGPT-User 13%0%5%10%15%20%'23'24'25'26
Table view
MonthChatGPT-UserSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20237%143
Oct 20238%143
Nov 20239%143
Dec 20239%143
Jan 202410%143
Feb 202410%143
Mar 202411%143
Apr 202413%143
May 202413%143
Jun 202412%143
Jul 202412%143
Aug 202413%143
Sep 202410%143
Oct 202410%143
Nov 202410%143
Dec 202411%143
Jan 202510%143
Feb 202510%143
Mar 202511%143
Apr 202510%143
May 202512%143
Jun 202510%143
Jul 202511%143
Aug 202510%143
Sep 202512%143
Oct 202512%143
Nov 202512%143
Dec 202513%143
Jan 202613%143
Feb 202613%143
Mar 202613%143
Apr 202613%143
May 202613%143
Jun 202613%143
Jul 202613%143
Aug 202614%143
Sep 202613%143
Oct 202613%143

Blocked by name (13)

AmazonYelpThe New York TimesBBCCNNNPRUSA TodayPoliticoThe VergeThe EconomistLinkedInTikTokQuora

Blocked by the general rules for all crawlers (8)

NextdoorRedditXFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Bloomberg

Named and allowed (15)

Best BuyeBayTemuInstacartAirbnbExpediaKAYAKTripadvisorVrboIndeedThe Wall Street JournalReutersThe AtlanticStack OverflowCVS Health

Recent changes

  • Oct 2026
    Bloomberg Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.
    Less blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Sep 2026
    WebMD Now allowed: ChatGPT-User.
    Less blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Dec 2025
    Amazon Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Sep 2025
    Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Aug 2025
    Glassdoor Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.
    Less blocking
  • Jun 2025
    Craigslist Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.
    Less blocking
  • May 2025
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking