{"domain":"nytimes.com","name":"The New York Times","category":"news","subcategory":"national","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 16 of the 24 bots in total. It names every major AI bot one by one. The homepage also showed our bot, which says who it is, a DataDome challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.111Z","last_checked":"2026-10-19T13:33:35.820Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.nytimes.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.nytimes.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.nytimes.com/robots.txt","hops":[{"from":"https://nytimes.com/robots.txt","to":"https://www.nytimes.com/robots.txt","status":301}],"blob":"04dee26f145ee77839c65ff45a4e1b887ae06be06b3b91fbc5043dc37277fcea","text":"# New York Times content is made available for your personal, non-commercial\n# use subject to our Terms of Service here:\n# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.\n# Use of any device, tool, or process designed to data mine or scrape the content\n# using automated means is prohibited without prior written permission from\n# The New York Times Company.  Prohibited uses include but are not limited to:\n# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in\n# the Digital Single Market;\n# (2) the development of any software, machine learning, artificial intelligence (AI),\n# and/or large language models (LLMs);\n# (3) creating or providing archived or cached data sets containing our content to others; and/or\n# (4) any commercial purposes.\n# Contact https://nytlicensing.com/contact/ for assistance.\n\nUser-agent: *\nUser-agent: Googlebot\nDisallow: /ads/\nDisallow: /adx/bin/\nDisallow: /athletic/wp/wp-admin/\nAllow: /athletic/wp/wp-admin/admin-ajax.php\nDisallow: /athletic/async-*\nDisallow: /athletic/search/*\nAllow: /athletic/search/$\nDisallow: /athletic/checkout/\nDisallow: /athletic/checkout?plan_id*\nAllow: /athletic/checkout/$\nDisallow: /athletic/checkout2*\nDisallow: /athletic/login/\nDisallow: /athletic/login?login_source*\nDisallow: /athletic/login?ref_page*\nAllow: /athletic/login/$\nDisallow: /athletic/login2/\nDisallow: /athletic/login2?login_source*\nDisallow: /athletic/login2?ref_page*\nAllow: /athletic/login2/$\nDisallow: /athletic/report/\nDisallow: /athletic/*/discuss/*\nAllow: /athletic/live-blogs/discuss/\nAllow: /athletic/nfl/game/discuss/\nDisallow: /athletic/register/\nDisallow: /athletic/register?welcome_redirect*\nDisallow: /athletic/register2/\nDisallow: /athletic/register2?welcome_redirect*\nDisallow: /athletic/betmgm-redirect*\nDisallow: /athletic/cdn-cgi/\nDisallow: /athletic/verizon/*\nDisallow: /athletic/forgot-password/*\nDisallow: /athletic/forgot-password2/*\nDisallow: /athletic/amp-social-login*\nDisallow: /athletic/track-analytics/\nDisallow: /athletic/amp-auth/\nDisallow: /athletic/rss-feed/\nDisallow: /athletic/*?*rss=1\nDisallow: /athletic/global-color-test.php\nDisallow: /athletic/global-font-test.php\nDisallow: /athletic/graphql*\nDisallow: /athletic/api*\nDisallow: /athletic/ip*\nDisallow: /athletic/call-set-cookie-with-context/*\nDisallow: /athletic/get-current-user/\nDisallow: /athletic/pv.json\nDisallow: /athletic/following-feed-test/*\nDisallow: /athletic*/boxscore/*\nDisallow: /athletic/feed-test/\nDisallow: /athletic*/signed-mp3-redirect-url/*\nDisallow: /athletic/embedded-interactive/*\nDisallow: /card/panel/\nDisallow: /panel/\nDisallow: /puzzles/leaderboards/invite/*\nDisallow: /svc\nAllow: /svc/crosswords\nAllow: /svc/games\nAllow: /svc/letter-boxed\nAllow: /svc/spelling-bee\nAllow: /svc/wordle\nAllow: /svc/connections\nAllow: /svc/sudoku\nAllow: /svc/strands\nAllow: /svc/pips\nDisallow: /video/embedded/*\nDisallow: /search\nDisallow: /multiproduct/\nDisallow: /hd/\nDisallow: /inyt/\nDisallow: /*?*query=\nDisallow: /*.pdf$\nDisallow: /*?*login=\nDisallow: /*?*campaignId=\nDisallow: /*?*mcubz=\nDisallow: /*?*smprod=\nDisallow: /*?*ProfileID=\nDisallow: /*?*ListingID=\nDisallow: /*?*campaign_id=\nDisallow: /*?*hybrid=\nDisallow: /*?*entry=\nDisallow: /*?*embed=\nDisallow: /*?ls=\nDisallow: /*?*&ls=\nDisallow: /v1/initialize*\nDisallow: /v1/rgstr*\nDisallow: /wirecutter/wp-admin/\nDisallow: /wirecutter/*.zip$\nDisallow: /wirecutter/*.csv$\nDisallow: /wirecutter/deals/beta\nDisallow: /wirecutter/data-requests\nDisallow: /wirecutter/search\nDisallow: /wirecutter/*?s=\nDisallow: /wirecutter/*&xid=\nDisallow: /wirecutter/*?q=\nDisallow: /wirecutter/*?l=\nDisallow: /wirecutter/*?merchant=\nDisallow: /wirecutter/out/\nDisallow: /search\nDisallow: /subscription/*?*source=\nDisallow: /subscription/*?*onboarded=\nDisallow: /*?*smid=\nDisallow: /*?*partner=\nDisallow: /*?*utm_source=\nAllow: /wirecutter/*?*utm_source=\nAllow: /ads/public/\nAllow: /svc/news/v3/all/pshb.rss\nAllow: /wirecutter/reviews/*?*utm_source=\nAllow: /wirecutter/blog/*?*utm_source=\nAllow: /wirecutter/lists/*?*utm_source=\nAllow: /wirecutter/gifts/*?*utm_source=\n\n\n# Googlebot Specific Rules\n\nUser-agent: Googlebot\nDisallow: /athletic*adgroupid*\nDisallow: /athletic*campaignid*\nDisallow: /athletic*ad_id*\nDisallow: /athletic*access_token*\nDisallow: /athletic*amp_reader_id*\nDisallow: /athletic*/?source=*\nDisallow: /athletic/*?*embed=1\n\n\n# Disallow Rules\n\nUser-agent: AliyunSecBot\nDisallow: /\n\nUser-agent: Amazonbot\nDisallow: /wirecutter/\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: archive.org_bot\nDisallow: /\n\nUser-agent: AudigentAdBot\nDisallow: /\n\nUser-agent: AwarioRssBot\nUser-agent: AwarioSmartBot\nDisallow: /\n\nUser-agent: BLEXBot\nDisallow: /\n\nUser-agent: Brightbot\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: ChatGPT-User\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: Claude-User\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: DataForSeoBot\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: Diffbot-User\nDisallow: /\n\nUser-agent: DuckAssistBot\nDisallow: /\n\nUser-agent: EchoboxBot\nDisallow: /\n\nUser-agent: ExaBot\nDisallow: /\n\nUser-agent: ExaSearchBot\nDisallow: /\n\nUser-agent: FacebookBot\nDisallow: /\n\nUser-agent: FirecrawlAgent\nDisallow: /\n\nUser-agent: FriendlyCrawler\nDisallow: /\n\nUser-agent: Google-CloudVertexBot\nDisallow: /\nAllow: /wirecutter/\nAllow: /athletic/\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: ImagesiftBot\nDisallow: /\n\nUser-agent: Jetslide\nDisallow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: Meta-ExternalAgent\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: Meta-ExternalFetcher\nUser-agent: meta-externalfetcher\nDisallow: /\n\nUser-agent: Meta-WebIndexer\nUser-agent: meta-webindexer/1.1\nDisallow: /\n\nUser-agent: MyCentralAIScraperBot\nDisallow: /\n\nUser-agent: NewsNow\nDisallow: /\n\nUser-agent: news-please\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: omgilibot\nDisallow: /\n\nUser-agent: peer39_crawler\nUser-agent: peer39_crawler/1.0\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: Poseidon Research Crawler\nDisallow: /\n\nUser-agent: quillbot.com\nDisallow: /\n\nUser-agent: Quora-Bot\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: SeekrBot\nDisallow: /\n\nUser-agent: SeznamHomepageCrawler\nDisallow: /\n\nUser-agent: ShapBot\nDisallow: /\n\nUser-agent: TaraGroup Intelligent Bot\nDisallow: /\n\nUser-agent: Timpibot\nDisallow: /\n\nUser-agent: TurnitinBot\nDisallow: /\n\nUser-agent: ViennaTinyBot\nDisallow: /\n\nUser-agent: YouBot\nDisallow: /\n\n\n# Ad Bot Rules\n\nUser-agent: AmazonAdBot\nAllow: /\n\n\n# Social Bot Rules\n\nUser-agent: facebookexternalhit\nAllow: /*?*smid=\n\nUser-agent: Twitterbot\nAllow: /*?*smid=\n\nUser-agent: RedditBot\nAllow: /*?*smid=\n\n# Sitemaps\n\nSitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/sitemap.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/collections.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/video.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/cooking.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/regions.xml\nSitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml\nSitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml\nSitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/espanol.xml.gz\nSitemap: https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz\nSitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-authors.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-verticals.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-teams.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-cities.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-players.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-tags.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-stats.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-schedule.xml\nSitemap: https://www.nytimes.com/athletic/sitemap-roster.xml\nSitemap: https://www.nytimes.com/athletic/sitemap.xml\nSitemap: https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":4,"text":"Use of any device, tool, or process designed to data mine or scrape the content"},{"line":9,"text":"(2) the development of any software, machine learning, artificial intelligence (AI),"},{"line":10,"text":"and/or large language models (LLMs);"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Googlebot","AliyunSecBot","Amazonbot","anthropic-ai","Applebot-Extended","archive.org_bot","AudigentAdBot","AwarioRssBot","AwarioSmartBot","BLEXBot","Brightbot","Bytespider","CCBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","Claude-Web","cohere-ai","DataForSeoBot","Diffbot","Diffbot-User","DuckAssistBot","EchoboxBot","ExaBot","ExaSearchBot","FacebookBot","FirecrawlAgent","FriendlyCrawler","Google-CloudVertexBot","Google-Extended","GPTBot","ImagesiftBot","Jetslide","magpie-crawler","Meta-ExternalAgent","meta-externalagent","Meta-ExternalFetcher","meta-externalfetcher","Meta-WebIndexer","meta-webindexer/1.1","MyCentralAIScraperBot","NewsNow","news-please","OAI-SearchBot","omgili","omgilibot","peer39_crawler","peer39_crawler/1.0","PerplexityBot","Perplexity-User","Poseidon Research Crawler","quillbot.com","Quora-Bot","Scrapy","SeekrBot","SeznamHomepageCrawler","ShapBot","TaraGroup Intelligent Bot","Timpibot","TurnitinBot","ViennaTinyBot","YouBot","AmazonAdBot","facebookexternalhit","Twitterbot","RedditBot"],"lines":365},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":233},"any":{"allowed":false,"line":233},"search":{"allowed":false,"line":233},"cart":{"allowed":false,"line":233},"checkout":{"allowed":false,"line":233},"account":{"allowed":false,"line":233}},"disallows":1,"lines":[232,233],"groupRanges":[[232,233]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":180},"any":{"allowed":false,"line":180},"search":{"allowed":false,"line":180},"cart":{"allowed":false,"line":180},"checkout":{"allowed":false,"line":180},"account":{"allowed":false,"line":180}},"disallows":1,"lines":[179,180],"groupRanges":[[179,180]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":230},"any":{"allowed":false,"line":230},"search":{"allowed":false,"line":230},"cart":{"allowed":false,"line":230},"checkout":{"allowed":false,"line":230},"account":{"allowed":false,"line":230}},"disallows":1,"lines":[229,230],"groupRanges":[[229,230]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":152},"any":{"allowed":false,"line":152},"search":{"allowed":false,"line":152},"cart":{"allowed":false,"line":152},"checkout":{"allowed":false,"line":152},"account":{"allowed":false,"line":152}},"disallows":1,"lines":[151,152],"groupRanges":[[151,152]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":174},"any":{"allowed":false,"line":174},"search":{"allowed":false,"line":174},"cart":{"allowed":false,"line":174},"checkout":{"allowed":false,"line":174},"account":{"allowed":false,"line":174}},"disallows":1,"lines":[173,174],"groupRanges":[[173,174]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":246},"any":{"allowed":false,"line":246},"search":{"allowed":false,"line":246},"cart":{"allowed":false,"line":246},"checkout":{"allowed":false,"line":246},"account":{"allowed":false,"line":246}},"disallows":1,"lines":[244,246],"groupRanges":[[244,246]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":171},"any":{"allowed":false,"line":171},"search":{"allowed":false,"line":171},"cart":{"allowed":false,"line":171},"checkout":{"allowed":false,"line":171},"account":{"allowed":false,"line":171}},"disallows":1,"lines":[170,171],"groupRanges":[[170,171]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":149},"any":{"allowed":false,"line":149},"search":{"allowed":false,"line":149},"cart":{"allowed":false,"line":149},"checkout":{"allowed":false,"line":149},"account":{"allowed":false,"line":149}},"disallows":1,"lines":[148,149],"groupRanges":[[148,149]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":266},"any":{"allowed":false,"line":266},"search":{"allowed":false,"line":266},"cart":{"allowed":false,"line":266},"checkout":{"allowed":false,"line":266},"account":{"allowed":false,"line":266}},"disallows":1,"lines":[265,266],"groupRanges":[[265,266]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":183},"any":{"allowed":false,"line":183},"search":{"allowed":false,"line":183},"cart":{"allowed":false,"line":183},"checkout":{"allowed":false,"line":183},"account":{"allowed":false,"line":183}},"disallows":1,"lines":[182,183],"groupRanges":[[182,183]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":279},"any":{"allowed":false,"line":279},"search":{"allowed":false,"line":279},"cart":{"allowed":false,"line":279},"checkout":{"allowed":false,"line":279},"account":{"allowed":false,"line":279}},"disallows":1,"lines":[278,279],"groupRanges":[[278,279]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":1,"lines":[145,146],"groupRanges":[[145,146]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":204},"any":{"allowed":false,"line":204},"search":{"allowed":false,"line":204},"cart":{"allowed":false,"line":204},"checkout":{"allowed":false,"line":204},"account":{"allowed":false,"line":204}},"disallows":1,"lines":[203,204],"groupRanges":[[203,204]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":254},"any":{"allowed":false,"line":254},"search":{"allowed":false,"line":254},"cart":{"allowed":false,"line":254},"checkout":{"allowed":false,"line":254},"account":{"allowed":false,"line":254}},"disallows":1,"lines":[252,254],"groupRanges":[[252,254]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":177},"any":{"allowed":false,"line":177},"search":{"allowed":false,"line":177},"cart":{"allowed":false,"line":177},"checkout":{"allowed":false,"line":177},"account":{"allowed":false,"line":177}},"disallows":1,"lines":[176,177],"groupRanges":[[176,177]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":186},"any":{"allowed":false,"line":186},"search":{"allowed":false,"line":186},"cart":{"allowed":false,"line":186},"checkout":{"allowed":false,"line":186},"account":{"allowed":false,"line":186}},"disallows":1,"lines":[185,186],"groupRanges":[[185,186]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":282},"any":{"allowed":false,"line":282},"search":{"allowed":false,"line":282},"cart":{"allowed":false,"line":282},"checkout":{"allowed":false,"line":282},"account":{"allowed":false,"line":282}},"disallows":1,"lines":[281,282],"groupRanges":[[281,282]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":250},"any":{"allowed":false,"line":250},"search":{"allowed":false,"line":250},"cart":{"allowed":false,"line":250},"checkout":{"allowed":false,"line":250},"account":{"allowed":false,"line":250}},"disallows":1,"lines":[248,250],"groupRanges":[[248,250]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":93,"lines":[15,125,130,137],"groupRanges":[[15,125],[130,137]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":81},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":86,"lines":[15,125],"groupRanges":[[15,125]]}],"lanes":{"training":{"level":"mixed","none":7,"full":2,"restricted":0,"n":9},"search":{"level":"mixed","none":5,"full":3,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":4,"full":3,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"DataDome challenge page"},"edge":{"cdn":["DataDome"],"botManagement":[{"vendor":"DataDome","cookies":["datadome","datadome"]}]},"llms":{"state":"absent","http":404,"url":"https://www.nytimes.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.nytimes.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.nytimes.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-08","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2025-09","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2025-10","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2025-11","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2025-12","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-01","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-02","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-03","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-04","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-05","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-06","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-07","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-08","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-09","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-10","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16}]},"events":[{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2026-04","precision":"month","source":"archive","provisional":false,"between":["2026-03-01","2026-04-01"],"summary":"Now blocked: Meta-WebIndexer.","dir":"up","up":["Meta-WebIndexer"],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20260301002328/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20260401001930/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /athletic/embedded-interactive/*"},{"t":"del","s":"Disallow: /athletic/interactive/*"},{"t":"ctx","s":"Disallow: /card/panel/"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AmazonBot"},{"t":"add","s":"Disallow: /wirecutter/"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: archive.org_bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: AudigentAdBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Meta-WebIndexer"},{"t":"add","s":"User-agent: meta-webindexer/1.1"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: MyCentralAIScraperBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Other Bot Rules"},{"t":"ctx","s":""},{"t":"add","s":"# Ad Bot Rules"},{"t":"add","s":""},{"t":"add","s":"User-agent: AmazonAdBot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":""},{"t":"add","s":"# Social Bot Rules"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: facebookexternalhit"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: RedditBot"},{"t":"add","s":"Allow: /*?*smid="},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml"},{"t":"add","s":"Sitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xml"},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/athletic/sitemap-authors.xml"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2025-10","precision":"month","source":"archive","provisional":false,"between":["2025-09-01","2025-10-01"],"summary":"Now allowed: Amazonbot.","dir":"down","up":[],"down":["Amazonbot"],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20250901041400/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20251001000520/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /athletic/forgot-password/*"},{"t":"del","s":"Allow: /athletic/forgot-password/$"},{"t":"ctx","s":"Disallow: /athletic/forgot-password2/*"},{"t":"del","s":"Allow: /athletic/forgot-password2/$"},{"t":"ctx","s":"Disallow: /athletic/amp-social-login*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /wirecutter/*?merchant="},{"t":"add","s":"Disallow: /wirecutter/out/"},{"t":"ctx","s":"Disallow: /search"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /svc/news/v3/all/pshb.rss"},{"t":"add","s":"Allow: /wirecutter/reviews/*?*utm_source="},{"t":"add","s":"Allow: /wirecutter/blog/*?*utm_source="},{"t":"add","s":"Allow: /wirecutter/lists/*?*utm_source="},{"t":"add","s":"Allow: /wirecutter/gifts/*?*utm_source="},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Amazonbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /wirecutter/"},{"t":"add","s":"Allow: /athletic/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml"},{"t":"add","s":"Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml"},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":""}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-07-01","2025-08-01"],"summary":"Now blocked: Claude-SearchBot, Claude-User and Perplexity-User.","dir":"up","up":["Claude-SearchBot","Claude-User","Perplexity-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"blocked","evidence":["https://web.archive.org/web/20250701001955/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20250801012859/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /athletic*/signed-mp3-redirect-url/*"},{"t":"add","s":"Disallow: /athletic/embedded-interactive/*"},{"t":"add","s":"Disallow: /athletic/interactive/*"},{"t":"ctx","s":"Disallow: /card/panel/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /svc/spelling-bee"},{"t":"del","s":"Allow: /svc/vertex"},{"t":"ctx","s":"Allow: /svc/wordle"},{"t":"add","s":"Allow: /svc/connections"},{"t":"add","s":"Allow: /svc/sudoku"},{"t":"add","s":"Allow: /svc/strands"},{"t":"add","s":"Allow: /svc/pips"},{"t":"ctx","s":"Disallow: /video/embedded/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AliyunSecBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AudigentAdBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: AwarioRssBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: BLEXBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Bytespider"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: EchoboxBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Jetslide"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: MyCentralAIScraperBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: NewsNow"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"gap","s":"… 39 more lines"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2025-04","precision":"month","source":"archive","provisional":false,"between":["2025-03-01","2025-04-01"],"summary":"Now blocked: DuckAssistBot and Meta-ExternalFetcher.","dir":"up","up":["DuckAssistBot","Meta-ExternalFetcher"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250301013739/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20250401005535/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /athletic/rss-feed/"},{"t":"add","s":"Disallow: /athletic/*?*rss=1"},{"t":"ctx","s":"Disallow: /athletic/global-color-test.php"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /athletic/feed-test/"},{"t":"add","s":"Disallow: /athletic*/signed-mp3-redirect-url/*"},{"t":"ctx","s":"Disallow: /card/panel/"},{"t":"add","s":"Disallow: /panel/"},{"t":"ctx","s":"Disallow: /puzzles/leaderboards/invite/*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /wirecutter/*?l="},{"t":"add","s":"Disallow: /wirecutter/*?merchant="},{"t":"ctx","s":"Disallow: /search"},{"t":"add","s":"Disallow: /subscription/*?*source="},{"t":"add","s":"Disallow: /subscription/*?*onboarded="},{"t":"ctx","s":"Disallow: /*?*smid="},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /athletic*/?source=*"},{"t":"del","s":"Disallow: /athletic/live-blogs/*embed=1*"},{"t":"add","s":"Disallow: /athletic/*?*embed=1"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: FriendlyCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-CloudVertexBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /wirecutter/"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ImagesiftBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Meta-ExternalFetcher"},{"t":"add","s":"User-agent: meta-externalfetcher"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: NewsNow"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Quora-Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Scrapy"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Timpibot"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 19 more lines"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-06-01","2024-08-01"],"summary":"Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.","dir":"up","up":["Applebot-Extended","Meta-ExternalAgent","OAI-SearchBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240601000016/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20240801002227/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":"User-agent: *"},{"t":"add","s":"User-agent: Googlebot"},{"t":"ctx","s":"Disallow: /ads/"},{"t":"ctx","s":"Disallow: /adx/bin/"},{"t":"add","s":"Disallow: /athletic/wp/wp-admin/"},{"t":"add","s":"Allow: /athletic/wp/wp-admin/admin-ajax.php"},{"t":"add","s":"Disallow: /athletic/async-*"},{"t":"add","s":"Disallow: /athletic/search/*"},{"t":"add","s":"Allow: /athletic/search/$"},{"t":"add","s":"Disallow: /athletic/checkout/"},{"t":"add","s":"Disallow: /athletic/checkout?plan_id*"},{"t":"add","s":"Allow: /athletic/checkout/$"},{"t":"add","s":"Disallow: /athletic/checkout2*"},{"t":"add","s":"Disallow: /athletic/login/"},{"t":"add","s":"Disallow: /athletic/login?login_source*"},{"t":"add","s":"Disallow: /athletic/login?ref_page*"},{"t":"add","s":"Allow: /athletic/login/$"},{"t":"add","s":"Disallow: /athletic/login2/"},{"t":"add","s":"Disallow: /athletic/login2?login_source*"},{"t":"add","s":"Disallow: /athletic/login2?ref_page*"},{"t":"add","s":"Allow: /athletic/login2/$"},{"t":"add","s":"Disallow: /athletic/report/"},{"t":"add","s":"Disallow: /athletic/*/discuss/*"},{"t":"add","s":"Disallow: /athletic/register/"},{"t":"add","s":"Disallow: /athletic/register?welcome_redirect*"},{"t":"add","s":"Disallow: /athletic/register2/"},{"t":"add","s":"Disallow: /athletic/register2?welcome_redirect*"},{"t":"add","s":"Disallow: /athletic/betmgm-redirect*"},{"t":"add","s":"Disallow: /athletic/cdn-cgi/"},{"t":"add","s":"Disallow: /athletic/verizon/*"},{"t":"add","s":"Disallow: /athletic/forgot-password/*"},{"t":"add","s":"Allow: /athletic/forgot-password/$"},{"t":"add","s":"Disallow: /athletic/forgot-password2/*"},{"t":"add","s":"Allow: /athletic/forgot-password2/$"},{"t":"add","s":"Disallow: /athletic/amp-social-login*"},{"t":"add","s":"Disallow: /athletic/track-analytics/"},{"t":"add","s":"Disallow: /athletic/amp-auth/"},{"t":"add","s":"Disallow: /athletic/rss-feed/"},{"t":"add","s":"Disallow: /athletic/global-color-test.php"},{"t":"add","s":"Disallow: /athletic/global-font-test.php"},{"t":"add","s":"Disallow: /athletic/graphql*"},{"t":"add","s":"Disallow: /athletic/api*"},{"t":"add","s":"Disallow: /athletic/ip*"},{"t":"add","s":"Disallow: /athletic/call-set-cookie-with-context/*"},{"t":"add","s":"Disallow: /athletic/get-current-user/"},{"t":"add","s":"Disallow: /athletic/pv.json"},{"t":"add","s":"Disallow: /athletic/following-feed-test/*"},{"t":"add","s":"Disallow: /athletic*/boxscore/*"},{"t":"add","s":"Disallow: /athletic/feed-test/"},{"t":"add","s":"Disallow: /card/panel/"},{"t":"ctx","s":"Disallow: /puzzles/leaderboards/invite/*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*?*ListingID="},{"t":"add","s":"Disallow: /*?*campaign_id="},{"t":"add","s":"Disallow: /*?*hybrid="},{"t":"add","s":"Disallow: /*?*entry="},{"t":"add","s":"Disallow: /*?*embed="},{"t":"add","s":"Disallow: /*?ls="},{"t":"add","s":"Disallow: /*?*&ls="},{"t":"ctx","s":"Disallow: /wirecutter/wp-admin/"},{"t":"gap","s":"… 83 more lines"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-02-01","2024-03-01"],"summary":"Now blocked: ClaudeBot, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.","dir":"up","up":["ClaudeBot","anthropic-ai","PerplexityBot","Amazonbot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240201000030/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20240301002928/https://www.nytimes.com/robots.txt"],"diff":[{"t":"add","s":"# New York Times content is made available for your personal, non-commercial"},{"t":"add","s":"# use subject to our Terms of Service here:"},{"t":"add","s":"# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service."},{"t":"add","s":"# Use of any device, tool, or process designed to data mine or scrape the content"},{"t":"add","s":"# using automated means is prohibited without prior written permission from"},{"t":"add","s":"# The New York Times Company.  Prohibited uses include but are not limited to:"},{"t":"add","s":"# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in"},{"t":"add","s":"# the Digital Single Market;"},{"t":"add","s":"# (2) the development of any software, machine learning, artificial intelligence (AI),"},{"t":"add","s":"# and/or large language models (LLMs);"},{"t":"add","s":"# (3) creating or providing archived or cached data sets containing our content to others; and/or"},{"t":"add","s":"# (4) any commercial purposes."},{"t":"add","s":"# Contact https://nytlicensing.com/contact/ for assistance."},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AwarioRssBot"},{"t":"add","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Bytespider"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ia_archiver"},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-Agent: omgili"},{"t":"add","s":"User-agent: omgili"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-Agent: omgilibot"},{"t":"add","s":"User-agent: omgilibot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"gap","s":"… 8 more lines"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2024-01","precision":"month","source":"archive","provisional":false,"between":["2023-12-01","2024-01-01"],"summary":"Now blocked: Bytespider.","dir":"up","up":["Bytespider"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231201004027/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20240101000929/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: CCBot"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001000841/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20231101000406/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: GPTBot"}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20230801000228/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20230901000459/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: Twitterbot"},{"t":"del","s":"Allow: /*?*smid="},{"t":"del","s":""},{"t":"del","s":"User-Agent: omgilibot"},{"t":"add","s":"User-agent: CCBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-Agent: omgili"},{"t":"add","s":"User-agent: GPTBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: CCBot"},{"t":"add","s":"User-Agent: omgili"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-Agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Twitterbot"},{"t":"add","s":"Allow: /*?*smid="},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml"},{"t":"add","s":""}]},{"domain":"nytimes.com","name":"The New York Times","category":"news","date":"2023-05","precision":"month","source":"archive","provisional":false,"between":["2023-04-01","2023-05-01"],"summary":"Now blocked: CCBot.","dir":"up","up":["CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230401000237/https://www.nytimes.com/robots.txt","https://web.archive.org/web/20230501002921/https://www.nytimes.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz"}]}]}