All sites / News and publishing
The New York Times
nytimes.comBlocks AI
Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 16 of the 24 bots in total. It names every major AI bot one by one. The homepage also showed our bot, which says who it is, a DataDome challenge page instead of the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
7 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
5 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
4 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 232 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 179 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 229 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 151 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 173 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 244 | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 170 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 148 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 265 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 182 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 278 | Suggests yes |
| AmazonbotAmazon | Allowed | Named, line 145 | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Named, line 203 | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Blocked | Named, line 252 | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 176 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 185 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 281 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 248 | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Claude-Web | Blocked | line 188 | Not in our verified list |
| cohere-ai | Blocked | line 191 | Not in our verified list |
| Diffbot | Blocked | line 197 | Documented by its operator |
| Diffbot-User | Blocked | line 200 | Documented by its operator |
| FacebookBot | Blocked | line 215 | Not in our verified list |
| FirecrawlAgent | Blocked | line 218 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 224 | Documented by its operator |
| ImagesiftBot | Blocked | line 235 | Not in our verified list |
| omgili | Blocked | line 268 | Not in our verified list |
| omgilibot | Blocked | line 271 | Not in our verified list |
| Timpibot | Blocked | line 308 | Not in our verified list |
| YouBot | Blocked | line 317 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# New York Times content is made available for your personal, non-commercial# use subject to our Terms of Service here:# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.# Use of any device, tool, or process designed to data mine or scrape the content# using automated means is prohibited without prior written permission from# The New York Times Company. Prohibited uses include but are not limited to:# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in# the Digital Single Market;# (2) the development of any software, machine learning, artificial intelligence (AI),# and/or large language models (LLMs);# (3) creating or providing archived or cached data sets containing our content to others; and/or# (4) any commercial purposes.# Contact https://nytlicensing.com/contact/ for assistance. User-agent: *User-agent: GooglebotDisallow: /ads/Disallow: /adx/bin/Disallow: /athletic/wp/wp-admin/Allow: /athletic/wp/wp-admin/admin-ajax.phpDisallow: /athletic/async-*Disallow: /athletic/search/*Allow: /athletic/search/$Disallow: /athletic/checkout/Disallow: /athletic/checkout?plan_id*Allow: /athletic/checkout/$Disallow: /athletic/checkout2*Disallow: /athletic/login/Disallow: /athletic/login?login_source*Disallow: /athletic/login?ref_page*Allow: /athletic/login/$Disallow: /athletic/login2/Disallow: /athletic/login2?login_source*Disallow: /athletic/login2?ref_page*Allow: /athletic/login2/$Disallow: /athletic/report/Disallow: /athletic/*/discuss/*Allow: /athletic/live-blogs/discuss/Allow: /athletic/nfl/game/discuss/Disallow: /athletic/register/Disallow: /athletic/register?welcome_redirect*Disallow: /athletic/register2/Disallow: /athletic/register2?welcome_redirect*Disallow: /athletic/betmgm-redirect*Disallow: /athletic/cdn-cgi/Disallow: /athletic/verizon/*Disallow: /athletic/forgot-password/*Disallow: /athletic/forgot-password2/*Disallow: /athletic/amp-social-login*Disallow: /athletic/track-analytics/Disallow: /athletic/amp-auth/Disallow: /athletic/rss-feed/Disallow: /athletic/*?*rss=1Disallow: /athletic/global-color-test.phpDisallow: /athletic/global-font-test.phpDisallow: /athletic/graphql*Disallow: /athletic/api*Disallow: /athletic/ip*Disallow: /athletic/call-set-cookie-with-context/*Disallow: /athletic/get-current-user/Disallow: /athletic/pv.jsonDisallow: /athletic/following-feed-test/*Disallow: /athletic*/boxscore/*Disallow: /athletic/feed-test/Disallow: /athletic*/signed-mp3-redirect-url/*Disallow: /athletic/embedded-interactive/*Disallow: /card/panel/Disallow: /panel/Disallow: /puzzles/leaderboards/invite/*Disallow: /svcAllow: /svc/crosswordsAllow: /svc/gamesAllow: /svc/letter-boxedAllow: /svc/spelling-beeAllow: /svc/wordleAllow: /svc/connectionsAllow: /svc/sudokuAllow: /svc/strandsAllow: /svc/pipsDisallow: /video/embedded/*Disallow: /searchDisallow: /multiproduct/Disallow: /hd/Disallow: /inyt/Disallow: /*?*query=Disallow: /*.pdf$Disallow: /*?*login=Disallow: /*?*campaignId=Disallow: /*?*mcubz=Disallow: /*?*smprod=Disallow: /*?*ProfileID=Disallow: /*?*ListingID=Disallow: /*?*campaign_id=Disallow: /*?*hybrid=Disallow: /*?*entry=Disallow: /*?*embed=Disallow: /*?ls=Disallow: /*?*&ls=Disallow: /v1/initialize*Disallow: /v1/rgstr*Disallow: /wirecutter/wp-admin/Disallow: /wirecutter/*.zip$Disallow: /wirecutter/*.csv$Disallow: /wirecutter/deals/betaDisallow: /wirecutter/data-requestsDisallow: /wirecutter/searchDisallow: /wirecutter/*?s=Disallow: /wirecutter/*&xid=Disallow: /wirecutter/*?q=Disallow: /wirecutter/*?l=Disallow: /wirecutter/*?merchant=Disallow: /wirecutter/out/Disallow: /searchDisallow: /subscription/*?*source=Disallow: /subscription/*?*onboarded=Disallow: /*?*smid=Disallow: /*?*partner=Disallow: /*?*utm_source=Allow: /wirecutter/*?*utm_source=Allow: /ads/public/Allow: /svc/news/v3/all/pshb.rssAllow: /wirecutter/reviews/*?*utm_source=Allow: /wirecutter/blog/*?*utm_source=Allow: /wirecutter/lists/*?*utm_source=Allow: /wirecutter/gifts/*?*utm_source= # Googlebot Specific Rules User-agent: GooglebotDisallow: /athletic*adgroupid*Disallow: /athletic*campaignid*Disallow: /athletic*ad_id*Disallow: /athletic*access_token*Disallow: /athletic*amp_reader_id*Disallow: /athletic*/?source=*Disallow: /athletic/*?*embed=1 # Disallow Rules User-agent: AliyunSecBotDisallow: /… 143 lines not about AI bots User-agent: AmazonbotDisallow: /wirecutter/ User-agent: anthropic-aiDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: archive.org_botDisallow: / User-agent: AudigentAdBotDisallow: / User-agent: AwarioRssBotUser-agent: AwarioSmartBotDisallow: / User-agent: BLEXBotDisallow: / User-agent: BrightbotDisallow: /… 15 lines not about AI bots User-agent: BytespiderDisallow: / User-agent: CCBotDisallow: / User-agent: ChatGPT-UserDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Claude-SearchBotDisallow: / User-agent: Claude-UserDisallow: / User-agent: Claude-WebDisallow: / User-agent: cohere-aiDisallow: / User-agent: DataForSeoBotDisallow: /… 2 lines not about AI bots User-agent: DiffbotDisallow: / User-agent: Diffbot-UserDisallow: / User-agent: DuckAssistBotDisallow: / User-agent: EchoboxBotDisallow: / User-agent: ExaBotDisallow: / User-agent: ExaSearchBotDisallow: /… 8 lines not about AI bots User-agent: FacebookBotDisallow: / User-agent: FirecrawlAgentDisallow: / User-agent: FriendlyCrawlerDisallow: /… 2 lines not about AI bots User-agent: Google-CloudVertexBotDisallow: /Allow: /wirecutter/Allow: /athletic/ User-agent: Google-ExtendedDisallow: / User-agent: GPTBotDisallow: / User-agent: ImagesiftBotDisallow: / User-agent: JetslideDisallow: / User-agent: magpie-crawlerDisallow: /… 5 lines not about AI bots User-agent: Meta-ExternalAgentUser-agent: meta-externalagentDisallow: / User-agent: Meta-ExternalFetcherUser-agent: meta-externalfetcherDisallow: / User-agent: Meta-WebIndexerUser-agent: meta-webindexer/1.1Disallow: / User-agent: MyCentralAIScraperBotDisallow: / User-agent: NewsNowDisallow: / User-agent: news-pleaseDisallow: /… 8 lines not about AI bots User-agent: OAI-SearchBotDisallow: / User-agent: omgiliDisallow: / User-agent: omgilibotDisallow: / User-agent: peer39_crawlerUser-agent: peer39_crawler/1.0Disallow: /… 3 lines not about AI bots User-agent: PerplexityBotDisallow: / User-agent: Perplexity-UserDisallow: / User-agent: Poseidon Research CrawlerDisallow: / User-agent: quillbot.comDisallow: / User-agent: Quora-BotDisallow: / User-agent: ScrapyDisallow: / User-agent: SeekrBotDisallow: / User-agent: SeznamHomepageCrawlerDisallow: / User-agent: ShapBotDisallow: / User-agent: TaraGroup Intelligent BotDisallow: /… 23 lines not about AI bots User-agent: TimpibotDisallow: / User-agent: TurnitinBotDisallow: / User-agent: ViennaTinyBotDisallow: /… 5 lines not about AI bots User-agent: YouBotDisallow: / # Ad Bot Rules User-agent: AmazonAdBotAllow: / # Social Bot Rules User-agent: facebookexternalhitAllow: /*?*smid= User-agent: TwitterbotAllow: /*?*smid= User-agent: RedditBotAllow: /*?*smid= # Sitemaps Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/sitemap.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/collections.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/video.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/cooking.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/regions.xmlSitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xmlSitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xmlSitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/espanol.xml.gzSitemap: https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gzSitemap: https://www.nytimes.com/wirecutter/sitemapindex.xmlSitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xmlSitemap: https://www.nytimes.com/athletic/sitemap-authors.xmlSitemap: https://www.nytimes.com/athletic/sitemap-verticals.xmlSitemap: https://www.nytimes.com/athletic/sitemap-teams.xmlSitemap: https://www.nytimes.com/athletic/sitemap-cities.xmlSitemap: https://www.nytimes.com/athletic/sitemap-players.xmlSitemap: https://www.nytimes.com/athletic/sitemap-tags.xmlSitemap: https://www.nytimes.com/athletic/sitemap-stats.xmlSitemap: https://www.nytimes.com/athletic/sitemap-schedule.xmlSitemap: https://www.nytimes.com/athletic/sitemap-roster.xmlSitemap: https://www.nytimes.com/athletic/sitemap.xmlSitemap: https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml … 46 lines not about AI bots
Notes about AI written in the file: "Use of any device, tool, or process designed to data mine or scrape the content" · "(2) the development of any software, machine learning, artificial intelligence (AI)," · "and/or large language models (LLMs);"
Everything else we checked
| Front door | Human check The homepage showed our bot, which says who it is, a DataDome challenge page instead of the site (HTTP 403).GET https://www.nytimes.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Page tags | Homepage not read. |
| Anti-bot software | Served through DataDome. The homepage set a DataDome cookie (datadome, datadome), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Apr 2026More blocking Now blocked: Meta-WebIndexer.Found in the Internet Archive's copies: changed between 1 Mar 2026 and 1 Apr 2026. Evidence: before · after
Show exactly what changed
Disallow: /athletic/embedded-interactive/*- Disallow: /athletic/interactive/*Disallow: /card/panel/…+ User-agent: AmazonBot+ Disallow: /wirecutter/+User-agent: anthropic-ai…+ User-agent: archive.org_bot+ Disallow: /+User-agent: AudigentAdBot…+ User-agent: Meta-WebIndexer+ User-agent: meta-webindexer/1.1+ Disallow: /+User-agent: MyCentralAIScraperBot…- # Other Bot Rules+ # Ad Bot Rules++ User-agent: AmazonAdBot+ Allow: /+++ # Social Bot Rules+User-agent: facebookexternalhit…+ User-agent: RedditBot+ Allow: /*?*smid=…Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml+ Sitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xmlSitemap: https://www.nytimes.com/athletic/sitemap-authors.xml - Oct 2025Less blocking Now allowed: Amazonbot.Found in the Internet Archive's copies: changed between 1 Sep 2025 and 1 Oct 2025. Evidence: before · after
Show exactly what changed
Disallow: /athletic/forgot-password/*- Allow: /athletic/forgot-password/$Disallow: /athletic/forgot-password2/*- Allow: /athletic/forgot-password2/$Disallow: /athletic/amp-social-login*…Disallow: /wirecutter/*?merchant=+ Disallow: /wirecutter/out/Disallow: /search…Allow: /svc/news/v3/all/pshb.rss+ Allow: /wirecutter/reviews/*?*utm_source=+ Allow: /wirecutter/blog/*?*utm_source=+ Allow: /wirecutter/lists/*?*utm_source=+ Allow: /wirecutter/gifts/*?*utm_source=…- User-agent: Amazonbot- Disallow: /-User-agent: anthropic-ai…Allow: /wirecutter/+ Allow: /athletic/…Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml+ Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xmlSitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz…- - Aug 2025More blocking Now blocked: Claude-SearchBot, Claude-User and Perplexity-User.Found in the Internet Archive's copies: changed between 1 Jul 2025 and 1 Aug 2025. Evidence: before · after
Show exactly what changed
Disallow: /athletic*/signed-mp3-redirect-url/*+ Disallow: /athletic/embedded-interactive/*+ Disallow: /athletic/interactive/*Disallow: /card/panel/…Allow: /svc/spelling-bee- Allow: /svc/vertexAllow: /svc/wordle+ Allow: /svc/connections+ Allow: /svc/sudoku+ Allow: /svc/strands+ Allow: /svc/pipsDisallow: /video/embedded/*…+ User-agent: AliyunSecBot+ Disallow: /+User-agent: Amazonbot…+ User-agent: AudigentAdBot+ Disallow: /+User-agent: AwarioRssBot…+ User-agent: BLEXBot+ Disallow: /+User-agent: Bytespider…+ User-agent: Claude-SearchBot+ Disallow: /++ User-agent: Claude-User+ Disallow: /+User-agent: Claude-Web…+ User-agent: EchoboxBot+ Disallow: /+User-agent: FacebookBot…+ User-agent: Jetslide+ Disallow: /+User-agent: magpie-crawler…+ User-agent: MyCentralAIScraperBot+ Disallow: /+User-agent: NewsNow…… 39 more lines - Apr 2025More blocking Now blocked: DuckAssistBot and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Mar 2025 and 1 Apr 2025. Evidence: before · after
Show exactly what changed
Disallow: /athletic/rss-feed/+ Disallow: /athletic/*?*rss=1Disallow: /athletic/global-color-test.php…Disallow: /athletic/feed-test/+ Disallow: /athletic*/signed-mp3-redirect-url/*Disallow: /card/panel/+ Disallow: /panel/Disallow: /puzzles/leaderboards/invite/*…Disallow: /wirecutter/*?l=+ Disallow: /wirecutter/*?merchant=Disallow: /search+ Disallow: /subscription/*?*source=+ Disallow: /subscription/*?*onboarded=Disallow: /*?*smid=…Disallow: /athletic*/?source=*- Disallow: /athletic/live-blogs/*embed=1*+ Disallow: /athletic/*?*embed=1…+ User-agent: DuckAssistBot+ Disallow: /+User-agent: FacebookBot…+ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: Google-CloudVertexBot+ Disallow: /+ Allow: /wirecutter/+User-agent: Google-Extended…+ User-agent: ImagesiftBot+ Disallow: /+User-agent: magpie-crawler…+ User-agent: Meta-ExternalFetcher+ User-agent: meta-externalfetcher+ Disallow: /+User-agent: NewsNow…+ User-agent: Quora-Bot+ Disallow: /+User-agent: Scrapy…+ User-agent: Timpibot+ Disallow: /… 19 more lines - Aug 2024More blocking Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.Found in the Internet Archive's copies: changed between 1 Jun 2024 and 1 Aug 2024. Evidence: before · after
Show exactly what changed
User-agent: *+ User-agent: GooglebotDisallow: /ads/Disallow: /adx/bin/+ Disallow: /athletic/wp/wp-admin/+ Allow: /athletic/wp/wp-admin/admin-ajax.php+ Disallow: /athletic/async-*+ Disallow: /athletic/search/*+ Allow: /athletic/search/$+ Disallow: /athletic/checkout/+ Disallow: /athletic/checkout?plan_id*+ Allow: /athletic/checkout/$+ Disallow: /athletic/checkout2*+ Disallow: /athletic/login/+ Disallow: /athletic/login?login_source*+ Disallow: /athletic/login?ref_page*+ Allow: /athletic/login/$+ Disallow: /athletic/login2/+ Disallow: /athletic/login2?login_source*+ Disallow: /athletic/login2?ref_page*+ Allow: /athletic/login2/$+ Disallow: /athletic/report/+ Disallow: /athletic/*/discuss/*+ Disallow: /athletic/register/+ Disallow: /athletic/register?welcome_redirect*+ Disallow: /athletic/register2/+ Disallow: /athletic/register2?welcome_redirect*+ Disallow: /athletic/betmgm-redirect*+ Disallow: /athletic/cdn-cgi/+ Disallow: /athletic/verizon/*+ Disallow: /athletic/forgot-password/*+ Allow: /athletic/forgot-password/$+ Disallow: /athletic/forgot-password2/*+ Allow: /athletic/forgot-password2/$+ Disallow: /athletic/amp-social-login*+ Disallow: /athletic/track-analytics/+ Disallow: /athletic/amp-auth/+ Disallow: /athletic/rss-feed/+ Disallow: /athletic/global-color-test.php+ Disallow: /athletic/global-font-test.php+ Disallow: /athletic/graphql*+ Disallow: /athletic/api*+ Disallow: /athletic/ip*+ Disallow: /athletic/call-set-cookie-with-context/*+ Disallow: /athletic/get-current-user/+ Disallow: /athletic/pv.json+ Disallow: /athletic/following-feed-test/*+ Disallow: /athletic*/boxscore/*+ Disallow: /athletic/feed-test/+ Disallow: /card/panel/Disallow: /puzzles/leaderboards/invite/*…Disallow: /*?*ListingID=+ Disallow: /*?*campaign_id=+ Disallow: /*?*hybrid=+ Disallow: /*?*entry=+ Disallow: /*?*embed=+ Disallow: /*?ls=+ Disallow: /*?*&ls=Disallow: /wirecutter/wp-admin/… 83 more lines - Mar 2024More blocking Now blocked: ClaudeBot, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Feb 2024 and 1 Mar 2024. Evidence: before · after
Show exactly what changed
+ # New York Times content is made available for your personal, non-commercial+ # use subject to our Terms of Service here:+ # https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.+ # Use of any device, tool, or process designed to data mine or scrape the content+ # using automated means is prohibited without prior written permission from+ # The New York Times Company. Prohibited uses include but are not limited to:+ # (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in+ # the Digital Single Market;+ # (2) the development of any software, machine learning, artificial intelligence (AI),+ # and/or large language models (LLMs);+ # (3) creating or providing archived or cached data sets containing our content to others; and/or+ # (4) any commercial purposes.+ # Contact https://nytlicensing.com/contact/ for assistance.+User-agent: *…+ User-agent: Amazonbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ Disallow: /+User-agent: Bytespider…+ User-agent: ChatGPT-User+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: DataForSeoBot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /+User-agent: Google-Extended…- User-agent: ia_archiver+ User-agent: magpie-crawlerDisallow: /- User-Agent: omgili+ User-agent: omgiliDisallow: /- User-Agent: omgilibot+ User-agent: omgilibotDisallow: /… 8 more lines - Jan 2024
- Nov 2023
- Sep 2023More blocking Now blocked: GPTBot.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
Show exactly what changed
- User-agent: Twitterbot- Allow: /*?*smid=-- User-Agent: omgilibot+ User-agent: CCBotDisallow: /- User-Agent: omgili+ User-agent: GPTBotDisallow: /…- User-agent: CCBot+ User-Agent: omgiliDisallow: /+ User-Agent: omgilibot+ Disallow: /++ User-agent: Twitterbot+ Allow: /*?*smid=+Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz…Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml+ - May 2023
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.