The Walled Web Newsletter

AI bots we track / AI search

Who blocks PerplexityBot?

PerplexityBot is Perplexity's bot for AI search. Blocking it usually means the site stops showing up as a source in that company's AI answers. Perplexity implies it obeys robots.txt: "It is not used to crawl content for AI foundation models." Source

33of 159
sites block it from the whole site
26of 33
of those name it in robots.txt
0of 159
block it from part of the site
9of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking PerplexityBot, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

PerplexityBot 20%
0%10%20%30%2023202420252026PerplexityBot 20%0%10%20%30%'23'24'25'26
Table view
MonthPerplexityBotSites with a file
Jan 20234%143
Feb 20234%143
Mar 20234%143
Apr 20234%143
May 20234%143
Jun 20234%143
Jul 20234%143
Aug 20234%143
Sep 20234%143
Oct 20234%143
Nov 20234%143
Dec 20234%143
Jan 20244%143
Feb 20244%143
Mar 20247%143
Apr 20248%143
May 202410%143
Jun 202412%143
Jul 202413%143
Aug 202415%143
Sep 202415%143
Oct 202415%143
Nov 202415%143
Dec 202417%143
Jan 202515%143
Feb 202516%143
Mar 202517%143
Apr 202517%143
May 202517%143
Jun 202516%143
Jul 202517%143
Aug 202516%143
Sep 202517%143
Oct 202518%143
Nov 202517%143
Dec 202518%143
Jan 202619%143
Feb 202620%143
Mar 202620%143
Apr 202620%143
May 202620%143
Jun 202620%143
Jul 202620%143
Aug 202620%143
Sep 202620%143
Oct 202620%143

Blocked by name (26)

AmazoneBayInstacartYelpThe New York TimesThe Washington PostThe GuardianBBCCNNAssociated PressBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayPoliticoThe VergeWIREDThe EconomistFacebookInstagramLinkedInTikTokQuoraThreads

Blocked by the general rules for all crawlers (7)

NextdoorThe Wall Street JournalReutersRedditXPinterestIMDb

Blocked from part of the site (0)

None.

Named and allowed (9)

Best BuyTemuAirbnbExpediaTripadvisorVrboIndeedGitHubSupabase

Recent changes

  • Oct 2026
    Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Aug 2026
    Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Less blocking
  • Dec 2025 to Jul 2026
    Craigslist Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.
    Less blocking
  • Mar 2026
    Instacart Now blocked: PerplexityBot.
    More blocking
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Jan 2026
    eBay Now blocked: ClaudeBot, CCBot, anthropic-ai, PerplexityBot and Amazonbot.
    More blocking
  • Oct 2025
    TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
    More blocking
  • Sep 2025
    Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
    More blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Aug 2025
    Glassdoor Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.
    Less blocking