{"domain":"theverge.com","name":"The Verge","category":"news","subcategory":"digital","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 15 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot and OAI-SearchBot can reach most of it.","computed_at":"2026-10-20T03:43:15.285Z","last_checked":"2026-10-19T13:36:09.735Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://theverge.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://theverge.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://theverge.com/robots.txt","hops":[],"blob":"f74664eb7fc3cbdfafd82167d05cb7b9596142a73509a3ce6f5ea83a0d1046a8","text":"\nUser-agent: Amazonbot\nAllow: /\n\nUser-agent: Applebot\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: A6-Indexer\nAllow: /sp/\nDisallow: /\n\nUser-agent: AhrefsBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: AI2Bot\nAllow: /sp/\nDisallow: /\n\nUser-agent: AI2Bot-DeepResearchEval\nAllow: /sp/\nDisallow: /\n\nUser-agent: Ai2Bot-Dolma\nAllow: /sp/\nDisallow: /\n\nUser-agent: aiHitBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Anchor Browser\nAllow: /sp/\nDisallow: /\n\nUser-agent: anthropic-ai\nAllow: /sp/\nDisallow: /\n\nUser-agent: ApifyBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: ApifyWebsiteContentCrawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: Applebot-Extended\nAllow: /sp/\nDisallow: /\n\nUser-agent: Baiduspider\nAllow: /sp/\nDisallow: /\n\nUser-agent: betaBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: BLEXBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: botify\nAllow: /sp/\nDisallow: /\n\nUser-agent: brands-bot-logo\nAllow: /sp/\nDisallow: /\n\nUser-agent: Brightbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Bytespider\nAllow: /sp/\nDisallow: /\n\nUser-agent: calculon\nAllow: /sp/\nDisallow: /\n\nUser-agent: CCBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: ChatGLM-Spider\nAllow: /sp/\nDisallow: /\n\nUser-agent: ChatGPT-User\nAllow: /sp/\nDisallow: /\n\nUser-agent: Claude-SearchBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Claude-User\nAllow: /sp/\nDisallow: /\n\nUser-agent: Claude-Web\nAllow: /sp/\nDisallow: /\n\nUser-agent: ClaudeBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: CloudVertexBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: cohere-ai\nAllow: /sp/\nDisallow: /\n\nUser-agent: cohere-training-data-crawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: collection@infegy.com\nAllow: /sp/\nDisallow: /\n\nUser-agent: Cotoyogi\nAllow: /sp/\nDisallow: /\n\nUser-agent: Crawl4AI\nAllow: /sp/\nDisallow: /\n\nUser-agent: CrunchBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: deepcrawl\nAllow: /sp/\nDisallow: /\n\nUser-agent: DeepSeekBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Diffbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: DirBuster\nAllow: /sp/\nDisallow: /\n\nUser-agent: DTS\nAllow: /sp/\nDisallow: /\n\nUser-agent: DuckAssistBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: EasouSpider\nAllow: /sp/\nDisallow: /\n\nUser-agent: FacebookBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: FirecrawlAgent\nAllow: /sp/\nDisallow: /\n\nUser-agent: FriendlyCrawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: GlutenCrawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: Google-CloudVertexBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Google-Extended\nAllow: /sp/\nDisallow: /\n\nUser-agent: GrapeshotCrawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: iaskspider\nAllow: /sp/\nDisallow: /\n\nUser-agent: ICC-Crawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: ImagesiftBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: img2dataset\nAllow: /sp/\nDisallow: /\n\nUser-agent: JackCrawl\nAllow: /sp/\nDisallow: /\n\nUser-agent: Jetslide\nAllow: /sp/\nDisallow: /\n\nUser-agent: linkapedia_bot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Mediatoolkitbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: MegaIndex\nAllow: /sp/\nDisallow: /\n\nUser-agent: meta-externalagent\nAllow: /sp/\nDisallow: /\n\nUser-agent: meta-externalfetcher\nAllow: /sp/\nDisallow: /\n\nUser-agent: Microsoft BITS\nAllow: /sp/\nDisallow: /\n\nUser-agent: Microsoft URL Control\nAllow: /sp/\nDisallow: /\n\nUser-agent: Microsoft-WebDAV-MiniRedir\nAllow: /sp/\nDisallow: /\n\nUser-agent: MistralAI-User\nAllow: /sp/\nDisallow: /\n\nUser-agent: news-please\nAllow: /sp/\nDisallow: /\n\nUser-agent: Novellum\nAllow: /sp/\nDisallow: /\n\nUser-agent: NTENTbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Nutch\nAllow: /sp/\nDisallow: /\n\nUser-agent: omgili\nAllow: /sp/\nDisallow: /\n\nUser-agent: Omgilibot\nAllow: /sp/\nDisallow: /\n\nUser-agent: OpenLinkProfiler\nAllow: /sp/\nDisallow: /\n\nUser-agent: Perplexity-User\nAllow: /sp/\nDisallow: /\n\nUser-agent: PerplexityBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: PetalBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Primalbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: python-requests\nAllow: /sp/\nDisallow: /\n\nUser-agent: Re-Animator\nAllow: /sp/\nDisallow: /\n\nUser-agent: Riddler\nAllow: /sp/\nDisallow: /\n\nUser-agent: SafeDNSBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: SBIntuitionsBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Scrapy\nAllow: /sp/\nDisallow: /\n\nUser-agent: SearchAtlas\nAllow: /sp/\nDisallow: /\n\nUser-agent: SemanticScholarBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: SemrushBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: SentiBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Sogou\nAllow: /sp/\nDisallow: /\n\nUser-agent: Spider-Pig\nAllow: /sp/\nDisallow: /\n\nUser-agent: sqlmap\nAllow: /sp/\nDisallow: /\n\nUser-agent: StorygizeBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Test Certificate Info\nAllow: /sp/\nDisallow: /\n\nUser-agent: TikTokSpider\nAllow: /sp/\nDisallow: /\n\nUser-agent: Timpibot\nAllow: /sp/\nDisallow: /\n\nUser-agent: Turnitin\nAllow: /sp/\nDisallow: /\n\nUser-agent: uipbot\nAllow: /sp/\nDisallow: /\n\nUser-agent: vBSEO\nAllow: /sp/\nDisallow: /\n\nUser-agent: VegeBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: VelenPublicWebCrawler\nAllow: /sp/\nDisallow: /\n\nUser-agent: Web Link Validator\nAllow: /sp/\nDisallow: /\n\nUser-agent: WebCruiser\nAllow: /sp/\nDisallow: /\n\nUser-agent: webzio-extended\nAllow: /sp/\nDisallow: /\n\nUser-agent: Xenu\nAllow: /sp/\nDisallow: /\n\nUser-agent: YouBot\nAllow: /sp/\nDisallow: /\n\nUser-agent: *\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n\nUser-agent: Googlebot-News\nDisallow: /ad/\nDisallow: /sponsored/\nDisallow: /sp/\n\nUser-agent: *\nAllow: /sp/\nDisallow: /admin\nDisallow: /newfanshot\nDisallow: /users/*/replies\nDisallow: /users/*/comments\nDisallow: /login\nDisallow: /account\nDisallow: /auth/*\nDisallow: /chorus_auth\nDisallow: /sso\nDisallow: /search\nDisallow: /the-highlight$\nDisallow: /share$\nDisallow: /share/*\nDisallow: /share?*\n\n# Google news sitemap\nSitemap: https://theverge.com/sitemaps/google_news\n\n# Sitemap archive\nSitemap: https://theverge.com/sitemaps\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["Amazonbot","Applebot","GPTBot","A6-Indexer","AhrefsBot","AI2Bot","AI2Bot-DeepResearchEval","Ai2Bot-Dolma","aiHitBot","Anchor Browser","anthropic-ai","ApifyBot","ApifyWebsiteContentCrawler","Applebot-Extended","Baiduspider","betaBot","BLEXBot","botify","brands-bot-logo","Brightbot","Bytespider","calculon","CCBot","ChatGLM-Spider","ChatGPT-User","Claude-SearchBot","Claude-User","Claude-Web","ClaudeBot","CloudVertexBot","cohere-ai","cohere-training-data-crawler","collection@infegy.com","Cotoyogi","Crawl4AI","CrunchBot","deepcrawl","DeepSeekBot","Diffbot","DirBuster","DTS","DuckAssistBot","EasouSpider","FacebookBot","FirecrawlAgent","FriendlyCrawler","GlutenCrawler","Google-CloudVertexBot","Google-Extended","GrapeshotCrawler","iaskspider","ICC-Crawler","ImagesiftBot","img2dataset","JackCrawl","Jetslide","linkapedia_bot","Mediatoolkitbot","MegaIndex","meta-externalagent","meta-externalfetcher","Microsoft BITS","Microsoft URL Control","Microsoft-WebDAV-MiniRedir","MistralAI-User","news-please","Novellum","NTENTbot","Nutch","omgili","Omgilibot","OpenLinkProfiler","Perplexity-User","PerplexityBot","PetalBot","Primalbot","python-requests","Re-Animator","Riddler","SafeDNSBot","SBIntuitionsBot","Scrapy","SearchAtlas","SemanticScholarBot","SemrushBot","SentiBot","Sogou","Spider-Pig","sqlmap","StorygizeBot","Test Certificate Info","TikTokSpider","Timpibot","Turnitin","uipbot","vBSEO","VegeBot","VelenPublicWebCrawler","Web Link Validator","WebCruiser","webzio-extended","Xenu","YouBot","*","Googlebot-News"],"lines":442},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":9},"any":{"allowed":true,"line":9},"search":{"allowed":true,"line":9},"cart":{"allowed":true,"line":9},"checkout":{"allowed":true,"line":9},"account":{"allowed":true,"line":9}},"disallows":0,"lines":[8,9],"groupRanges":[[8,9]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":113},"any":{"allowed":false,"line":113},"search":{"allowed":false,"line":113},"cart":{"allowed":false,"line":113},"checkout":{"allowed":false,"line":113},"account":{"allowed":false,"line":113}},"disallows":1,"lines":[111,113],"groupRanges":[[111,113]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":193},"any":{"allowed":false,"line":193},"search":{"allowed":false,"line":193},"cart":{"allowed":false,"line":193},"checkout":{"allowed":false,"line":193},"account":{"allowed":false,"line":193}},"disallows":1,"lines":[191,193],"groupRanges":[[191,193]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":53},"any":{"allowed":false,"line":53},"search":{"allowed":false,"line":53},"cart":{"allowed":false,"line":53},"checkout":{"allowed":false,"line":53},"account":{"allowed":false,"line":53}},"disallows":1,"lines":[51,53],"groupRanges":[[51,53]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":89},"any":{"allowed":false,"line":89},"search":{"allowed":false,"line":89},"cart":{"allowed":false,"line":89},"checkout":{"allowed":false,"line":89},"account":{"allowed":false,"line":89}},"disallows":1,"lines":[87,89],"groupRanges":[[87,89]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":237},"any":{"allowed":false,"line":237},"search":{"allowed":false,"line":237},"cart":{"allowed":false,"line":237},"checkout":{"allowed":false,"line":237},"account":{"allowed":false,"line":237}},"disallows":1,"lines":[235,237],"groupRanges":[[235,237]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":81},"any":{"allowed":false,"line":81},"search":{"allowed":false,"line":81},"cart":{"allowed":false,"line":81},"checkout":{"allowed":false,"line":81},"account":{"allowed":false,"line":81}},"disallows":1,"lines":[79,81],"groupRanges":[[79,81]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":21},"any":{"allowed":false,"line":21},"search":{"allowed":false,"line":21},"cart":{"allowed":false,"line":21},"checkout":{"allowed":false,"line":21},"account":{"allowed":false,"line":21}},"disallows":1,"lines":[19,21],"groupRanges":[[19,21]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":41},"any":{"allowed":false,"line":41},"search":{"allowed":false,"line":41},"cart":{"allowed":false,"line":41},"checkout":{"allowed":false,"line":41},"account":{"allowed":false,"line":41}},"disallows":1,"lines":[39,41],"groupRanges":[[39,41]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":101},"any":{"allowed":false,"line":101},"search":{"allowed":false,"line":101},"cart":{"allowed":false,"line":101},"checkout":{"allowed":false,"line":101},"account":{"allowed":false,"line":101}},"disallows":1,"lines":[99,101],"groupRanges":[[99,101]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":293},"any":{"allowed":false,"line":293},"search":{"allowed":false,"line":293},"cart":{"allowed":false,"line":293},"checkout":{"allowed":false,"line":293},"account":{"allowed":false,"line":293}},"disallows":1,"lines":[291,293],"groupRanges":[[291,293]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":0,"lines":[2,3],"groupRanges":[[2,3]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":165},"any":{"allowed":false,"line":165},"search":{"allowed":false,"line":165},"cart":{"allowed":false,"line":165},"checkout":{"allowed":false,"line":165},"account":{"allowed":false,"line":165}},"disallows":1,"lines":[163,165],"groupRanges":[[163,165]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":97},"any":{"allowed":false,"line":97},"search":{"allowed":false,"line":97},"cart":{"allowed":false,"line":97},"checkout":{"allowed":false,"line":97},"account":{"allowed":false,"line":97}},"disallows":1,"lines":[95,97],"groupRanges":[[95,97]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":105},"any":{"allowed":false,"line":105},"search":{"allowed":false,"line":105},"cart":{"allowed":false,"line":105},"checkout":{"allowed":false,"line":105},"account":{"allowed":false,"line":105}},"disallows":1,"lines":[103,105],"groupRanges":[[103,105]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":289},"any":{"allowed":false,"line":289},"search":{"allowed":false,"line":289},"cart":{"allowed":false,"line":289},"checkout":{"allowed":false,"line":289},"account":{"allowed":false,"line":289}},"disallows":1,"lines":[287,289],"groupRanges":[[287,289]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":241},"any":{"allowed":false,"line":241},"search":{"allowed":false,"line":241},"cart":{"allowed":false,"line":241},"checkout":{"allowed":false,"line":241},"account":{"allowed":false,"line":241}},"disallows":1,"lines":[239,241],"groupRanges":[[239,241]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":257},"any":{"allowed":false,"line":257},"search":{"allowed":false,"line":257},"cart":{"allowed":false,"line":257},"checkout":{"allowed":false,"line":257},"account":{"allowed":false,"line":257}},"disallows":1,"lines":[255,257],"groupRanges":[[255,257]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":431},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":false,"line":427}},"disallows":15,"lines":[411,413,420,435],"groupRanges":[[411,413],[420,435]]}],"lanes":{"training":{"level":"mixed","none":7,"full":2,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":5,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":5,"full":2,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":["Fastly"],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.theverge.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.theverge.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.theverge.com/.well-known/tdmrep.json"},"meta":{"title":"The Verge","robotsMeta":[{"name":"bingbot","content":"nocache"},{"name":"robots","content":"index,follow"}],"noai":[],"noindex":false,"jsonLdTypes":["NewsMediaOrganization","ImageObject","Person","Thing","WebSite","SearchAction","EntryPoint"],"platform":"WordPress","textRatio":0.021,"llmsLink":false},"x_robots_tag":null,"tos":{"url":"https://www.pmc.com/terms-of-use/","source":"discovered","skipped":"ToS hosted on another domain; not fetched in v0.1"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15}]},"events":[{"domain":"theverge.com","name":"The Verge","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-05-01","2026-06-04"],"summary":"Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.","dir":"up","up":["Meta-ExternalAgent","AI2Bot","Claude-SearchBot","DuckAssistBot","Claude-User","Perplexity-User","Meta-ExternalFetcher","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260501002446/https://www.theverge.com/robots.txt","https://web.archive.org/web/20260604075151/https://www.theverge.com/robots.txt"],"diff":[{"t":"del","s":"User-agent: *"},{"t":"del","s":"Disallow: /wp-admin/"},{"t":"del","s":"Allow: /wp-admin/admin-ajax.php"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: A6-Indexer"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Applebot-Extended"},{"t":"add","s":"User-agent: AhrefsBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Applebot"},{"t":"del","s":"Allow: /"},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AI2Bot-DeepResearchEval"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"add","s":"User-agent: aiHitBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Anchor Browser"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ApifyBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"add","s":"User-agent: ApifyWebsiteContentCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /sp/"},{"t":"gap","s":"… 383 more lines"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2025-05","precision":"month","source":"archive","provisional":false,"between":["2025-04-01","2025-05-01"],"summary":"Now allowed: Amazonbot.","dir":"down","up":[],"down":["Amazonbot"],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250401031222/https://www.theverge.com/robots.txt","https://web.archive.org/web/20250501054043/https://www.theverge.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: Amazonbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Applebot-Extended"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2025-02","precision":"month","source":"archive","provisional":false,"between":["2025-01-01","2025-02-01"],"summary":"Named GPTBot explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250101001114/https://www.theverge.com/robots.txt","https://web.archive.org/web/20250201073817/https://www.theverge.com/robots.txt"],"diff":[{"t":"del","s":""},{"t":"del","s":"User-agent: Googlebot-News"},{"t":"del","s":"Disallow: /admin"},{"t":"del","s":"Disallow: /newfanshot"},{"t":"del","s":"Disallow: /users/*/replies"},{"t":"del","s":"Disallow: /users/*/comments"},{"t":"del","s":"Disallow: /search"},{"t":"del","s":"Disallow: /account"},{"t":"del","s":"Disallow: /login"},{"t":"del","s":"Disallow: /chorus_auth"},{"t":"del","s":"Disallow: /sso"},{"t":"del","s":"Disallow: /ad"},{"t":"del","s":"Disallow: /sponsored"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"del","s":"Disallow: /admin"},{"t":"del","s":"Disallow: /newfanshot"},{"t":"del","s":"Disallow: /users/*/replies"},{"t":"del","s":"Disallow: /users/*/comments"},{"t":"del","s":"Disallow: /search"},{"t":"del","s":"Disallow: /account"},{"t":"del","s":"Disallow: /login"},{"t":"del","s":"Disallow: /chorus_auth"},{"t":"del","s":"Disallow: /sso"},{"t":"del","s":"Disallow: */archives/*/archives*"},{"t":"add","s":"Disallow: /wp-admin/"},{"t":"add","s":"Allow: /wp-admin/admin-ajax.php"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://www.theverge.com/sitemaps"},{"t":"del","s":"Sitemap: https://www.theverge.com/sitemaps/authors"},{"t":"del","s":"Sitemap: https://www.theverge.com/sitemaps/groups"},{"t":"del","s":"Sitemap: https://www.theverge.com/sitemaps/videos"},{"t":"add","s":"User-agent: Googlebot-News"},{"t":"add","s":"Disallow: /ad"},{"t":"add","s":"Disallow: /sponsored"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: *"},{"t":"add","s":"Disallow: /admin"},{"t":"add","s":"Disallow: /newfanshot"},{"t":"add","s":"Disallow: /users/*/replies"},{"t":"add","s":"Disallow: /users/*/comments"},{"t":"add","s":"Disallow: /login"},{"t":"add","s":"Disallow: /account/*"},{"t":"add","s":"Disallow: /auth/*"},{"t":"add","s":"Disallow: /chorus_auth"},{"t":"add","s":"Disallow: /sso"},{"t":"add","s":"Disallow: /search"},{"t":"add","s":"Disallow: /the-highlight$"},{"t":"gap","s":"… 12 more lines"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2024-07","precision":"month","source":"archive","provisional":false,"between":["2024-06-01","2024-07-01"],"summary":"Now blocked: Applebot-Extended. Now allowed: GPTBot.","dir":"mixed","up":["Applebot-Extended"],"down":["GPTBot"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240601001608/https://www.theverge.com/robots.txt","https://web.archive.org/web/20240701004615/https://www.theverge.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"ctx","s":"Disallow: /"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2024-05","precision":"month","source":"archive","provisional":false,"between":["2024-04-01","2024-05-01"],"summary":"Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.","dir":"up","up":["ClaudeBot","CCBot","Bytespider","anthropic-ai","PerplexityBot","Amazonbot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240401002248/https://www.theverge.com/robots.txt","https://web.archive.org/web/20240501102951/https://www.theverge.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /sso"},{"t":"add","s":"Disallow: */archives/*/archives*"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ImagesiftBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"gap","s":"… 7 more lines"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001000320/https://www.theverge.com/robots.txt","https://web.archive.org/web/20231101003911/https://www.theverge.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"}]},{"domain":"theverge.com","name":"The Verge","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801013521/https://www.theverge.com/robots.txt","https://web.archive.org/web/20230901030045/https://www.theverge.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"}]}]}