All sites / News and publishing
WIRED
wired.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 15 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot and ChatGPT-User can reach most of it.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
6 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
5 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
4 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 55 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 55 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 55 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 55 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 55 | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 55 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 55 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 55 | Suggests yes |
| AmazonbotAmazon | Blocked | Named, line 55 | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Named, line 55 | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Blocked | Named, line 55 | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 55 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 55 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 55 | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Blocked | Named, line 55 | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| cohere-ai | Blocked | line 65 | Not in our verified list |
| cohere-training-data-crawler | Blocked | line 66 | Not in our verified list |
| Diffbot | Blocked | line 67 | Documented by its operator |
| FirecrawlAgent | Blocked | line 70 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 71 | Documented by its operator |
| GoogleOther | Blocked | line 73 | Documented by its operator |
| PetalBot | Blocked | line 84 | Not in our verified list |
| Timpibot | Blocked | line 88 | Not in our verified list |
| YouBot | Blocked | line 90 | Not in our verified list |
| Webzio | Blocked | line 91 | Not in our verified list |
| Webzio-Extended | Blocked | line 92 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
Sitemap: https://www.wired.com/sitemap.xmlSitemap: https://www.wired.com/tagpages-sitemap.xmlSitemap: https://www.wired.com/contributors-sitemap.xmlSitemap: https://www.wired.com/bundles-sitemap.xmlSitemap: https://www.wired.com/branded-sitemap.xmlSitemap: https://www.wired.com/feed/google-latest-news/sitemap-google-newsSitemap: https://www.wired.com/feed/rss User-agent: *Disallow: /*?Allow: /*?pageAllow: /*.xml?Allow: /*.jsAllow: /*.cssAllow: /*.icoAllow: /verso/staticAllow: /*.svgAllow: /*.pngAllow: /*.jpgAllow: /*.jpegAllow: /*rss?Allow: /*?id=Disallow: /auth/Disallow: /account/Disallow: /user/Disallow: /user-contextDisallow: /preview/Disallow: /searchDisallow: /product/Disallow: /cdn-cgi/Disallow: /services.min.jsDisallow: /com.condenast/yv8Disallow: /reject-allDisallow: */c_limit/ ###Allow: /v2/offers/wiraAllow: /*?*utm_medium=socialAllow: /*?*utm_medium=syndicationAllow: /*?ref=Allow: /story/madison-square-garden-jim-dolan-surveillance-machine/?srcDisallow: */testing-products-### User-agent: AmazonAdBotUser-agent: FacebookExternalHitUser-agent: Google-InspectionToolUser-agent: LinkedInBotUser-agent: PinterestbotUser-agent: Storebot-GoogleUser-agent: TikTokSpiderUser-agent: TwitterbotAllow: /… 53 lines not about AI bots User-agent: AmazonbotUser-agent: Applebot-ExtendedUser-agent: archive.org_botUser-agent: BravebotUser-agent: Brightbot 1.0User-agent: BytespiderUser-agent: CCBotUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: ClaudeBotUser-agent: cohere-aiUser-agent: cohere-training-data-crawlerUser-agent: DiffbotUser-agent: DuckAssistBotUser-agent: factset_spyderbotUser-agent: FirecrawlAgentUser-agent: Google-CloudVertexBotUser-agent: Google-ExtendedUser-agent: GoogleOtherUser-agent: heritrixUser-agent: ia_archiverUser-agent: ia_archiver-web.archive.orgUser-agent: meta-externalagentUser-agent: meta-externalfetcherUser-agent: meta-webindexerUser-agent: MistralAI-UserUser-agent: PanguBotUser-agent: Perplexity-UserUser-agent: PerplexityBotUser-agent: PetalBotUser-agent: ScrapyUser-agent: Shap-UserUser-agent: ShapBotUser-agent: TimpibotUser-agent: YisouspiderUser-agent: YouBotUser-agent: WebzioUser-agent: Webzio-ExtendedDisallow: /
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.wired.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Page tags | <meta name="robots" content="index, follow, max-image-preview:large"> |
| Anti-bot software | Served through Cloudflare, Amazon CloudFront.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: Organization, ImageObject, WebSite, SearchAction, ItemList, ListItem, CollectionPage. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | https://www.condenast.com/user-agreement/ (ToS hosted on another domain; not fetched in v0.1) |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Oct 2025Mixed Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.Found in the Internet Archive's copies: changed between 1 Sep 2025 and 1 Oct 2025. Evidence: before · after
Show exactly what changed
Disallow: /partners/- Disallow: /sponsored/Disallow: /video/right_rail*…Disallow: /com.condenast/yv8*+ Disallow: */testing-products-…Disallow: /preview/+ Disallow: */testing-products-…Disallow: /services.min.js?_=+ Disallow: */testing-products-…- User-agent: CCBot- Disallow: /-User-agent: Google-Extended- Disallow: /-+ User-agent: Google-CloudVertexBot+ User-agent: GoogleOther+ User-agent: Applebot-ExtendedUser-agent: Amazonbot- Disallow: /-- User-agent: FacebookBot- Disallow: /-+ User-agent: meta-webindexerUser-agent: meta-externalagent- Disallow: /-- User-agent: anthropic-ai- Disallow: /-- User-agent: Claude-Web- Disallow: /-- User-agent: cohere-ai- Disallow: /-- User-agent: Omgilibot- Disallow: /-- User-agent: omgili- Disallow: /-+ User-agent: meta-externalfetcher+ User-agent: ClaudeBot+ User-agent: Claude-SearchBot+ User-agent: Claude-UserUser-agent: PerplexityBot- Disallow: /-- User-agent: YouBot- Disallow: /… 24 more lines - Nov 2024More blocking Now blocked: Meta-ExternalAgent.Found in the Internet Archive's copies: changed between 1 Oct 2024 and 1 Nov 2024. Evidence: before · after
Show exactly what changed
Disallow: /services.min.js?_=+ Disallow: /com.condenast/yv8*…+ User-agent: meta-externalagent+ Disallow: /+User-agent: anthropic-ai - May to Sep 2024Mixed Now blocked: Applebot-Extended and Bytespider. Now allowed: GPTBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 May 2024 and 1 Sep 2024. Evidence: before · after
Show exactly what changed
Disallow: /account/*+ Disallow: /preview/Allow: /wp-content/…Disallow: /services.min.js?_=+ Disallow: /preview/…- User-agent: GPTBot- Disallow: /-- User-agent: ChatGPT-User- Disallow: /-User-agent: Google-Extended…- User-Agent: PerplexityBot+ User-agent: omgiliDisallow: /+ User-agent: PerplexityBot+ Disallow: /++ User-agent: YouBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xmlSitemap: https://www.wired.com/branded-sitemap.xml- Sitemap: https://www.wired.com/coupons/sitemap.xml+ #Sitemap: https://www.wired.com/coupons/sitemap.xmlSitemap: https://www.wired.com/video/sitemap.xml - Mar 2024More blocking Now blocked: PerplexityBot.Found in the Internet Archive's copies: changed between 1 Feb 2024 and 1 Mar 2024. Evidence: before · after
Show exactly what changed
Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…+ User-Agent: PerplexityBot+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml…Sitemap: https://www.wired.com/sitemap-archive-1.xml+ Sitemap: https://www.wired.com/bundles-sitemap.xml - Dec 2023More blocking Now blocked: Google-Extended, anthropic-ai and Amazonbot.Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Dec 2023. Evidence: before · after
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml - Sep 2023More blocking Now blocked: GPTBot, CCBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
Show exactly what changed
Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/…Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/…Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/+ User-agent: Google-InspectionTool+ Allow: /++ User-agent: CCBot+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml…Sitemap: https://www.wired.com/contributors-sitemap.xml+ Sitemap: https://www.wired.com/categories-sitemap.xml?hierarchy=tags#Sitemap: https://www.wired.com/categories-sitemap.xml+ Sitemap: https://www.wired.com/sitemap-archive-1.xml
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.