The Walled Web Newsletter

AI bots we track / AI search

Who blocks OAI-SearchBot?

OAI-SearchBot is OpenAI's bot for AI search. Blocking it usually means the site stops showing up as a source in that company's AI answers. OpenAI says it obeys robots.txt: Controls inclusion in ChatGPT search results; changes take about 24 hours. Source

19of 159
sites block it from the whole site
11of 19
of those name it in robots.txt
1of 159
block it from part of the site
15of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking OAI-SearchBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

OAI-SearchBot 11%
0%5%10%15%20%2023202420252026OAI-SearchBot 11%0%5%10%15%20%'23'24'25'26
Table view
MonthOAI-SearchBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20244%143
Apr 20245%143
May 20244%143
Jun 20244%143
Jul 20244%143
Aug 20246%143
Sep 20245%143
Oct 20245%143
Nov 20246%143
Dec 20246%143
Jan 20256%143
Feb 20256%143
Mar 20256%143
Apr 20256%143
May 20258%143
Jun 20256%143
Jul 20258%143
Aug 20257%143
Sep 20258%143
Oct 20258%143
Nov 20258%143
Dec 20259%143
Jan 20268%143
Feb 20269%143
Mar 202610%143
Apr 202610%143
May 202610%143
Jun 202610%143
Jul 202610%143
Aug 202612%143
Sep 202612%143
Oct 202611%143

Blocked by name (11)

AmazonYelpThe New York TimesBBCCNNNPRUSA TodayPoliticoThe EconomistTikTokQuora

Blocked by the general rules for all crawlers (8)

NextdoorRedditXFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (1)

Bloomberg

Named and allowed (15)

Best BuyeBayTemuInstacartAirbnbExpediaTripadvisorVrboIndeedThe Wall Street JournalReutersThe AtlanticLinkedInGitHubHumana

Recent changes

  • Oct 2026
    Bloomberg Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.
    Less blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Aug 2026
    The Economist Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.
    More blocking
  • Jun 2026
    Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    More blocking
  • Feb to May 2026
    Pinterest Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Dec 2025
    Amazon Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.
    More blocking
  • Dec 2025
    LinkedIn Now allowed: OAI-SearchBot.
    Less blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Jul 2025
    NPR Now blocked: OAI-SearchBot.
    More blocking