The Verge

theverge.com

Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 15 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot and OAI-SearchBot can reach most of it.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

7 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

3 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Some blocked

5 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI	Allowed	Named, line 8	Says yes
ClaudeBotAnthropic	Blocked	Named, line 111	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Blocked	Named, line 191	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Blocked	Named, line 51	Says yes
CCBotCommon Crawl	Blocked	Named, line 87	Says yes
Meta-ExternalAgentMeta	Blocked	Named, line 235	Suggests yes
BytespiderByteDance	Blocked	Named, line 79	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Blocked	Named, line 19	Does not say
anthropic-aiAnthropic (legacy) · old name	Blocked	Named, line 39	Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Blocked	Named, line 99	Says yes
PerplexityBotPerplexity	Blocked	Named, line 291	Suggests yes
AmazonbotAmazon	Allowed	Named, line 2	Says yes
Amzn-SearchBotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Blocked	Named, line 163	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Some blocked
ChatGPT-UserOpenAI	Blocked	Named, line 95	Says not always
Claude-UserAnthropic	Blocked	Named, line 103	Says yes
Perplexity-UserPerplexity	Blocked	Named, line 287	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Blocked	Named, line 239	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Blocked	Named, line 255	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

Applebot	Has rules	line 5	Documented by its operator
AI2Bot-DeepResearchEval	Blocked	line 23	Not in our verified list
Ai2Bot-Dolma	Blocked	line 27	Not in our verified list
Claude-Web	Blocked	line 107	Not in our verified list
CloudVertexBot	Blocked	line 115	Not in our verified list
cohere-ai	Blocked	line 119	Not in our verified list
cohere-training-data-crawler	Blocked	line 123	Not in our verified list
DeepSeekBot	Blocked	line 147	Not in our verified list
Diffbot	Blocked	line 151	Documented by its operator
FacebookBot	Blocked	line 171	Not in our verified list
FirecrawlAgent	Blocked	line 175	Not in our verified list
Google-CloudVertexBot	Blocked	line 187	Documented by its operator
ImagesiftBot	Blocked	line 207	Not in our verified list
Novellum	Blocked	line 263	Not in our verified list
omgili	Blocked	line 275	Not in our verified list
Omgilibot	Blocked	line 279	Not in our verified list
PetalBot	Blocked	line 295	Not in our verified list
Timpibot	Blocked	line 367	Not in our verified list
webzio-extended	Blocked	line 399	Not in our verified list
YouBot	Blocked	line 407	Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 442 lines · 111 lines about AI bots Source

 User-agent: AmazonbotAllow: / User-agent: ApplebotAllow: / User-agent: GPTBotAllow: / User-agent: A6-IndexerAllow: /sp/Disallow: / User-agent: AhrefsBotAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: AI2BotAllow: /sp/Disallow: / User-agent: AI2Bot-DeepResearchEvalAllow: /sp/Disallow: / User-agent: Ai2Bot-DolmaAllow: /sp/Disallow: / User-agent: aiHitBotAllow: /sp/Disallow: / User-agent: Anchor BrowserAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: anthropic-aiAllow: /sp/Disallow: / User-agent: ApifyBotAllow: /sp/Disallow: / User-agent: ApifyWebsiteContentCrawlerAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: Applebot-ExtendedAllow: /sp/Disallow: / User-agent: BaiduspiderAllow: /sp/Disallow: / User-agent: betaBotAllow: /sp/Disallow: / User-agent: BLEXBotAllow: /sp/Disallow: / User-agent: botifyAllow: /sp/Disallow: / User-agent: brands-bot-logoAllow: /sp/Disallow: / User-agent: BrightbotAllow: /sp/Disallow: /… 23 lines not about AI bots User-agent: BytespiderAllow: /sp/Disallow: / User-agent: calculonAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: CCBotAllow: /sp/Disallow: / User-agent: ChatGLM-SpiderAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: ChatGPT-UserAllow: /sp/Disallow: / User-agent: Claude-SearchBotAllow: /sp/Disallow: / User-agent: Claude-UserAllow: /sp/Disallow: / User-agent: Claude-WebAllow: /sp/Disallow: / User-agent: ClaudeBotAllow: /sp/Disallow: / User-agent: CloudVertexBotAllow: /sp/Disallow: / User-agent: cohere-aiAllow: /sp/Disallow: / User-agent: cohere-training-data-crawlerAllow: /sp/Disallow: / User-agent: collection@infegy.comAllow: /sp/Disallow: / User-agent: CotoyogiAllow: /sp/Disallow: / User-agent: Crawl4AIAllow: /sp/Disallow: / User-agent: CrunchBotAllow: /sp/Disallow: / User-agent: deepcrawlAllow: /sp/Disallow: /… 19 lines not about AI bots User-agent: DeepSeekBotAllow: /sp/Disallow: / User-agent: DiffbotAllow: /sp/Disallow: / User-agent: DirBusterAllow: /sp/Disallow: / User-agent: DTSAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: DuckAssistBotAllow: /sp/Disallow: / User-agent: EasouSpiderAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: FacebookBotAllow: /sp/Disallow: / User-agent: FirecrawlAgentAllow: /sp/Disallow: / User-agent: FriendlyCrawlerAllow: /sp/Disallow: / User-agent: GlutenCrawlerAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: Google-CloudVertexBotAllow: /sp/Disallow: / User-agent: Google-ExtendedAllow: /sp/Disallow: / User-agent: GrapeshotCrawlerAllow: /sp/Disallow: / User-agent: iaskspiderAllow: /sp/Disallow: / User-agent: ICC-CrawlerAllow: /sp/Disallow: /… 11 lines not about AI bots User-agent: ImagesiftBotAllow: /sp/Disallow: / User-agent: img2datasetAllow: /sp/Disallow: / User-agent: JackCrawlAllow: /sp/Disallow: / User-agent: JetslideAllow: /sp/Disallow: / User-agent: linkapedia_botAllow: /sp/Disallow: / User-agent: MediatoolkitbotAllow: /sp/Disallow: / User-agent: MegaIndexAllow: /sp/Disallow: /… 23 lines not about AI bots User-agent: meta-externalagentAllow: /sp/Disallow: / User-agent: meta-externalfetcherAllow: /sp/Disallow: / User-agent: Microsoft BITSAllow: /sp/Disallow: / User-agent: Microsoft URL ControlAllow: /sp/Disallow: / User-agent: Microsoft-WebDAV-MiniRedirAllow: /sp/Disallow: /… 11 lines not about AI bots User-agent: MistralAI-UserAllow: /sp/Disallow: / User-agent: news-pleaseAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: NovellumAllow: /sp/Disallow: / User-agent: NTENTbotAllow: /sp/Disallow: / User-agent: NutchAllow: /sp/Disallow: /… 7 lines not about AI bots User-agent: omgiliAllow: /sp/Disallow: / User-agent: OmgilibotAllow: /sp/Disallow: / User-agent: OpenLinkProfilerAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: Perplexity-UserAllow: /sp/Disallow: / User-agent: PerplexityBotAllow: /sp/Disallow: / User-agent: PetalBotAllow: /sp/Disallow: / User-agent: PrimalbotAllow: /sp/Disallow: / User-agent: python-requestsAllow: /sp/Disallow: / User-agent: Re-AnimatorAllow: /sp/Disallow: / User-agent: RiddlerAllow: /sp/Disallow: / User-agent: SafeDNSBotAllow: /sp/Disallow: / User-agent: SBIntuitionsBotAllow: /sp/Disallow: / User-agent: ScrapyAllow: /sp/Disallow: / User-agent: SearchAtlasAllow: /sp/Disallow: / User-agent: SemanticScholarBotAllow: /sp/Disallow: / User-agent: SemrushBotAllow: /sp/Disallow: / User-agent: SentiBotAllow: /sp/Disallow: / User-agent: SogouAllow: /sp/Disallow: / User-agent: Spider-PigAllow: /sp/Disallow: / User-agent: sqlmapAllow: /sp/Disallow: / User-agent: StorygizeBotAllow: /sp/Disallow: / User-agent: Test Certificate InfoAllow: /sp/Disallow: / User-agent: TikTokSpiderAllow: /sp/Disallow: /… 67 lines not about AI bots User-agent: TimpibotAllow: /sp/Disallow: / User-agent: TurnitinAllow: /sp/Disallow: / User-agent: uipbotAllow: /sp/Disallow: / User-agent: vBSEOAllow: /sp/Disallow: / User-agent: VegeBotAllow: /sp/Disallow: / User-agent: VelenPublicWebCrawlerAllow: /sp/Disallow: / User-agent: Web Link ValidatorAllow: /sp/Disallow: / User-agent: WebCruiserAllow: /sp/Disallow: /… 27 lines not about AI bots User-agent: webzio-extendedAllow: /sp/Disallow: / User-agent: XenuAllow: /sp/Disallow: /… 3 lines not about AI bots User-agent: YouBotAllow: /sp/Disallow: / User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.php User-agent: Googlebot-NewsDisallow: /ad/Disallow: /sponsored/Disallow: /sp/ User-agent: *Allow: /sp/Disallow: /adminDisallow: /newfanshotDisallow: /users/*/repliesDisallow: /users/*/commentsDisallow: /loginDisallow: /accountDisallow: /auth/*Disallow: /chorus_authDisallow: /ssoDisallow: /searchDisallow: /the-highlight$Disallow: /share$Disallow: /share/*Disallow: /share?* # Google news sitemapSitemap: https://theverge.com/sitemaps/google_news # Sitemap archiveSitemap: https://theverge.com/sitemaps … 32 lines not about AI bots

Everything else we checked

Front door	Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://theverge.com/ as WalledWebBot, from a residential connection in the United States
Page tags	`<meta name="bingbot" content="nocache">` `<meta name="robots" content="index,follow">`
Anti-bot software	Served through Fastly.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	Structured data types: NewsMediaOrganization, ImageObject, Person, Thing, WebSite, SearchAction, EntryPoint. Most of the page is script; an agent that does not run JavaScript sees little text. Built on WordPress.Homepage only. Useful for site owners; not part of the result.
Terms of Service	https://www.pmc.com/terms-of-use/ (ToS hosted on another domain; not fetched in v0.1)
Not found	No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

47 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Jun 2026
More blocking Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
Found in the Internet Archive's copies: changed between 1 May 2026 and 4 Jun 2026. Evidence: before · after

Show exactly what changed
- User-agent: *
- Disallow: /wp-admin/
- Allow: /wp-admin/admin-ajax.php

+ User-agent: Amazonbot
+ Allow: /
+
+ User-agent: Applebot
+ Allow: /
+
User-agent: GPTBot
…

- User-agent: Google-Extended
+ User-agent: A6-Indexer
Disallow: /
+ Allow: /sp/

- User-agent: Applebot-Extended
+ User-agent: AhrefsBot
Disallow: /
+ Allow: /sp/

- User-agent: Applebot
- Allow: /
+ User-agent: AI2Bot
+ Disallow: /
+ Allow: /sp/

+ User-agent: AI2Bot-DeepResearchEval
+ Disallow: /
+ Allow: /sp/
+
+ User-agent: Ai2Bot-Dolma
+ Disallow: /
+ Allow: /sp/
+
+ User-agent: aiHitBot
+ Disallow: /
+ Allow: /sp/
+
+ User-agent: Anchor Browser
+ Disallow: /
+ Allow: /sp/
+
User-agent: anthropic-ai
Disallow: /
+ Allow: /sp/

+ User-agent: ApifyBot
+ Disallow: /
+ Allow: /sp/
+
+ User-agent: ApifyWebsiteContentCrawler
+ Disallow: /
+ Allow: /sp/
+
+ User-agent: Applebot-Extended
+ Disallow: /
+ Allow: /sp/
… 383 more lines
May 2025
Less blocking Now allowed: Amazonbot.
Found in the Internet Archive's copies: changed between 1 Apr 2025 and 1 May 2025. Evidence: before · after

Show exactly what changed

- User-agent: Amazonbot
- Disallow: /
-
User-agent: Applebot-Extended
Feb 2025
Signals Named GPTBot explicitly, without changing what it may reach.
Found in the Internet Archive's copies: changed between 1 Jan 2025 and 1 Feb 2025. Evidence: before · after

Show exactly what changed
-
- User-agent: Googlebot-News
- Disallow: /admin
- Disallow: /newfanshot
- Disallow: /users/*/replies
- Disallow: /users/*/comments
- Disallow: /search
- Disallow: /account
- Disallow: /login
- Disallow: /chorus_auth
- Disallow: /sso
- Disallow: /ad
- Disallow: /sponsored
-
User-agent: *
- Disallow: /admin
- Disallow: /newfanshot
- Disallow: /users/*/replies
- Disallow: /users/*/comments
- Disallow: /search
- Disallow: /account
- Disallow: /login
- Disallow: /chorus_auth
- Disallow: /sso
- Disallow: */archives/*/archives*
+ Disallow: /wp-admin/
+ Allow: /wp-admin/admin-ajax.php

+ User-agent: GPTBot
+ Allow: /
+
User-agent: Google-Extended
…

+ User-agent: Applebot
+ Allow: /
+
User-agent: anthropic-ai
…

- Sitemap: https://www.theverge.com/sitemaps
- Sitemap: https://www.theverge.com/sitemaps/authors
- Sitemap: https://www.theverge.com/sitemaps/groups
- Sitemap: https://www.theverge.com/sitemaps/videos
+ User-agent: Googlebot-News
+ Disallow: /ad
+ Disallow: /sponsored

+ User-agent: *
+ Disallow: /admin
+ Disallow: /newfanshot
+ Disallow: /users/*/replies
+ Disallow: /users/*/comments
+ Disallow: /login
+ Disallow: /account/*
+ Disallow: /auth/*
+ Disallow: /chorus_auth
+ Disallow: /sso
+ Disallow: /search
+ Disallow: /the-highlight$
… 12 more lines
Jul 2024
Mixed Now blocked: Applebot-Extended. Now allowed: GPTBot.
Found in the Internet Archive's copies: changed between 1 Jun 2024 and 1 Jul 2024. Evidence: before · after

Show exactly what changed

- User-agent: GPTBot
- Disallow: /
-
User-agent: Google-Extended
…

- User-agent: Applebot
+ User-agent: Applebot-Extended
Disallow: /
May 2024
More blocking Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.
Found in the Internet Archive's copies: changed between 1 Apr 2024 and 1 May 2024. Evidence: before · after

Show exactly what changed

- User-agent: GPTBot
- Disallow: /
-
- User-agent: Google-Extended
- Disallow: /
-
User-agent: *
…
Disallow: /sso
+ Disallow: */archives/*/archives*

+ User-agent: GPTBot
+ Disallow: /
+
+ User-agent: Google-Extended
+ Disallow: /
+
+ User-agent: Amazonbot
+ Disallow: /
+
+ User-agent: Applebot
+ Disallow: /
+
+ User-agent: anthropic-ai
+ Disallow: /
+
+ User-agent: Bytespider
+ Disallow: /
+
+ User-agent: CCBot
+ Disallow: /
+
+ User-agent: ChatGPT-User
+ Disallow: /
+
+ User-agent: ClaudeBot
+ Disallow: /
+
+ User-agent: Claude-Web
+ Disallow: /
+
+ User-agent: Diffbot
+ Disallow: /
+
+ User-agent: FacebookBot
+ Disallow: /
+
+ User-agent: ImagesiftBot
+ Disallow: /
+
+ User-agent: Omgilibot
+ Disallow: /
+
+ User-agent: Omgili
+ Disallow: /
+
+ User-agent: PerplexityBot
+ Disallow: /
+
… 7 more lines
Nov 2023
More blocking Now blocked: Google-Extended.
Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after

Show exactly what changed

+ User-agent: Google-Extended
+ Disallow: /
+
User-agent: *
Sep 2023
More blocking Now blocked: GPTBot.
Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after

Show exactly what changed

+ User-agent: GPTBot
+ Disallow: /
+
User-agent: *

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.