{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","subcategory":"business","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 21 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot and ChatGPT-User can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.143Z","last_checked":"2026-10-19T13:33:39.337Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://www.wsj.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.wsj.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.wsj.com/robots.txt","hops":[{"from":"https://wsj.com/robots.txt","to":"https://www.wsj.com/robots.txt","status":301}],"blob":"89210f23a6252c1f03f3615e35e81ab50714fcea36cb56e8ef7725fdb91de3aa","text":"# NOTICE: Collection of content and other data on https://www.wsj.com/ through\n# automated means is prohibited unless you have express written\n# permission from Dow Jones & Company, Inc. and may only be conducted for the\n# limited purpose contained in said permission.\n#\n# Dow Jones & Company, Inc. Terms of Use may be found at\n# https://www.dowjones.com/terms-of-use/\n#\n# If you would like to apply for permission to license the\n# intellectual property and/or other materials of Dow Jones & Company, Inc.’s\n# brands, please contact us via email at copyright@dowjones.com.\n\nUser-agent: *\nDisallow: /\n\nUser-agent: googlebot\nUser-agent: googlebot-image\nUser-agent: GoogleOther\nUser-agent: Googlebot-Video\nUser-agent: Google-InspectionTool\nUser-agent: AdsBot-Google\nUser-agent: AdsBot-Google-Mobile\nUser-agent: AdsBot-Google-Mobile-Apps\nUser-agent: Storebot-Google\nUser-agent: google-read-aloud\nUser-agent: bingbot\nUser-agent: msnbot\nUser-agent: bingpreview\nUser-agent: slurp\nUser-agent: yahoo\nUser-agent: baiduspider\nUser-agent: Pinterestbot\nUser-agent: Yeti\nUser-agent: MojeekBot\nUser-agent: 360Spider\nUser-agent: google-cloudvertexbot\nUser-agent: duckduckbot\nUser-agent: Applebot\nUser-agent: flipboard\nUser-agent: qwantbot\nUser-agent: SeznamBot\n\n\n\nUser-agent: proximic\nUser-agent: admantx\nUser-agent: thetradedesk\nUser-agent: outbrain\nUser-agent: ias_crawler\nUser-agent: AmazonAdBot\nUser-agent: pubmatic\nUser-agent: smartologybot\nUser-agent: parselybot\nUser-agent: BomboraBot\nUser-agent: IllumaBot/1.0\nUser-agent: Publication-Access-for-Facebook\nUser-agent: Wallie\nUser-agent: mediapartners-google\n\n\nUser-agent: Screaming Frog SEO Spider\nUser-agent: AhrefsBot\nUser-agent: SemrushBot\nUser-agent: SimilarWebBot\nUser-agent: SISTRIX\nUser-agent: botify\nUser-agent: Chrome-Lighthouse\n\n\n\nUser-agent: ChatGPT-User\nUser-agent: GPTBot\nUser-agent: OAI-SearchBot\n\n\nUser-agent: facebookexternalhit\nUser-agent: facebot\nUser-agent: twitterbot\nUser-agent: linkedinbot\nUser-agent: snapchat\nUser-agent: sentry\nUser-agent: Iframely\nUser-agent: Vocabtracker\nUser-agent: EpvzCrawl6194680250\nUser-agent: Citoid\nUser-agent: ZoteroTranslationServer\n\n\n\nAllow: /\nDisallow: /article_email/*\nDisallow: /user/*\nDisallow: /pdf/documents/*\nDisallow: /login/*\nDisallow: /acct/*\nDisallow: /msgcenter/*\nDisallow: /setup/*\nDisallow: /marketing/*\nDisallow: /public/article/*\nDisallow: /public/resources/documents/*\nDisallow: /public/search/\nDisallow: /public/search*\nDisallow: /search*\nDisallow: /public/page/wsj-x-marketing.html\nDisallow: /public/page/news-media-marketing.html\nDisallow: /public/page/0_0_WP_RT_MARKETING.html\nDisallow: /news/articles/SB2*\nDisallow: /news/articles/SB3*\nDisallow: /news/articles/SB4*\nDisallow: /articles/SB2*\nDisallow: /articles/SB3*\nDisallow: /articles/SB4*\nDisallow: /article/AP*\nDisallow: /article/BT-CO*\nDisallow: /article/DN-CO*\nDisallow: /article/PR-CO*\nDisallow: /article/HUG*\nDisallow: /video/search/*\nDisallow: /articles/BT-CO*\nDisallow: /articles/DN-CO*\nDisallow: /articles/PR-CO*\nDisallow: /news/articles/BT-CO*\nDisallow: /news/articles/DN-CO*\nDisallow: /news/articles/PR-CO*\nDisallow: /catchup/*\nDisallow: /articles/the-meaning-behind-juneteenth-11592413234\nDisallow: /emailservice/*\nDisallow: /emailsignup/*\nDisallow: /insetsrv/v1/*\nDisallow: /user/fpd/api/*\nDisallow: /Date(*\nDisallow: /auth/sso/proxy-login*\nDisallow: /client/\n\n# For Buyside Search Results\nDisallow: /buyside/search-results?*term=*\n# Don't crawl non-indexable sites\nDisallow: /*?type=mdc_*&id=*\nDisallow: /*?id=*&type=mdc_*\nDisallow: /market-data/quotes/*/options/*\nDisallow: /subscribe/?inttrackingCode=*\nDisallow: /subscribe/?template=*\n\n\nSitemap: https://www.wsj.com/sitemap.xml\nSitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml\nSitemap: https://www.wsj.com/wsj_video_recent.xml\nSitemap: https://www.wsj.com/sitemap_topics.xml\nSitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml\nSitemap: https://www.wsj.com/live_news_sitemap.xml\nSitemap: https://www.wsj.com/authors_sitemap.xml\nSitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml\nSitemap: https://www.wsj.com/buyside/sitemap.xml\nSitemap: https://www.wsj.com/wsj_quote_index_sitemap.xml\nSitemap: https://www.wsj.com/wsjsitemaps/wsj_recipes.xml\nSitemap: https://www.wsj.com/sitemap_topic_collections.xml\n\n\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["*","googlebot","googlebot-image","GoogleOther","Googlebot-Video","Google-InspectionTool","AdsBot-Google","AdsBot-Google-Mobile","AdsBot-Google-Mobile-Apps","Storebot-Google","google-read-aloud","bingbot","msnbot","bingpreview","slurp","yahoo","baiduspider","Pinterestbot","Yeti","MojeekBot","360Spider","google-cloudvertexbot","duckduckbot","Applebot","flipboard","qwantbot","SeznamBot","proximic","admantx","thetradedesk","outbrain","ias_crawler","AmazonAdBot","pubmatic","smartologybot","parselybot","BomboraBot","IllumaBot/1.0","Publication-Access-for-Facebook","Wallie","mediapartners-google","Screaming Frog SEO Spider","AhrefsBot","SemrushBot","SimilarWebBot","SISTRIX","botify","Chrome-Lighthouse","ChatGPT-User","GPTBot","OAI-SearchBot","facebookexternalhit","facebot","twitterbot","linkedinbot","snapchat","sentry","Iframely","Vocabtracker","EpvzCrawl6194680250","Citoid","ZoteroTranslationServer"],"lines":159},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":90},"any":{"allowed":true,"line":90},"search":{"allowed":false,"line":103},"cart":{"allowed":true,"line":90},"checkout":{"allowed":true,"line":90},"account":{"allowed":true,"line":90}},"disallows":49,"lines":[16,142],"groupRanges":[[16,142]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":90},"any":{"allowed":true,"line":90},"search":{"allowed":false,"line":103},"cart":{"allowed":true,"line":90},"checkout":{"allowed":true,"line":90},"account":{"allowed":true,"line":90}},"disallows":49,"lines":[16,142],"groupRanges":[[16,142]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":90},"any":{"allowed":true,"line":90},"search":{"allowed":false,"line":103},"cart":{"allowed":true,"line":90},"checkout":{"allowed":true,"line":90},"account":{"allowed":true,"line":90}},"disallows":49,"lines":[16,142],"groupRanges":[[16,142]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":14},"any":{"allowed":false,"line":14},"search":{"allowed":false,"line":14},"cart":{"allowed":false,"line":14},"checkout":{"allowed":false,"line":14},"account":{"allowed":false,"line":14}},"disallows":1,"lines":[13,14],"groupRanges":[[13,14]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":90},"any":{"allowed":true,"line":90},"search":{"allowed":false,"line":103},"cart":{"allowed":true,"line":90},"checkout":{"allowed":true,"line":90},"account":{"allowed":true,"line":90}},"disallows":49,"lines":[16,142],"groupRanges":[[16,142]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":90},"any":{"allowed":true,"line":90},"search":{"allowed":false,"line":103},"cart":{"allowed":true,"line":90},"checkout":{"allowed":true,"line":90},"account":{"allowed":true,"line":90}},"disallows":49,"lines":[16,142],"groupRanges":[[16,142]]}],"lanes":{"training":{"level":"mixed","none":8,"full":1,"restricted":0,"n":9},"search":{"level":"mixed","none":7,"full":1,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":6,"full":1,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":["Amazon CloudFront"],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":21},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":21},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":21},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":21},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":21}]},"events":[{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-05-01","2026-06-02"],"summary":"Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.","dir":"up","up":["MistralAI-Training","AI2Bot","Claude-SearchBot","Amazonbot","Amzn-SearchBot","DuckAssistBot","MistralAI-Index","Meta-WebIndexer","Claude-User","Perplexity-User","Google-Agent","Amzn-User","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260501000338/https://www.wsj.com/robots.txt","https://web.archive.org/web/20260602153104/https://www.wsj.com/robots.txt"],"diff":[{"t":"add","s":"# NOTICE: Collection of content and other data on https://www.wsj.com/ through"},{"t":"add","s":"# automated means is prohibited unless you have express written"},{"t":"add","s":"# permission from Dow Jones & Company, Inc. and may only be conducted for the"},{"t":"add","s":"# limited purpose contained in said permission."},{"t":"add","s":"#"},{"t":"add","s":"# Dow Jones & Company, Inc. Terms of Use may be found at"},{"t":"add","s":"# https://www.dowjones.com/terms-of-use/"},{"t":"add","s":"#"},{"t":"add","s":"# If you would like to apply for permission to license the"},{"t":"add","s":"# intellectual property and/or other materials of Dow Jones & Company, Inc.’s"},{"t":"add","s":"# brands, please contact us via email at copyright@dowjones.com."},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"del","s":"Sitemap: https://www.wsj.com/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/sitemap_topics.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/live_news_sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsj_graphics_sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_types.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/authors_sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp"},{"t":"del","s":"Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp"},{"t":"del","s":"Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/buyside/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: googlebot"},{"t":"add","s":"User-agent: googlebot-image"},{"t":"add","s":"User-agent: GoogleOther"},{"t":"add","s":"User-agent: Googlebot-Video"},{"t":"add","s":"User-agent: Google-InspectionTool"},{"t":"add","s":"User-agent: AdsBot-Google"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile-Apps"},{"t":"add","s":"User-agent: Storebot-Google"},{"t":"add","s":"User-agent: google-read-aloud"},{"t":"add","s":"User-agent: mediapartners-google"},{"t":"add","s":"User-agent: bingbot"},{"t":"add","s":"User-agent: msnbot"},{"t":"add","s":"User-agent: bingpreview"},{"t":"add","s":"User-agent: slurp"},{"t":"add","s":"User-agent: yahoo"},{"t":"add","s":"User-agent: baiduspider"},{"t":"add","s":"User-agent: Pinterestbot"},{"t":"add","s":"User-agent: Yeti"},{"t":"add","s":"User-agent: MojeekBot"},{"t":"add","s":"User-agent: 360Spider"},{"t":"add","s":"User-agent: google-cloudvertexbot"},{"t":"add","s":"User-agent: duckduckbot"},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: flipboard"},{"t":"add","s":"User-agent: qwantbot"},{"t":"add","s":"User-agent: SeznamBot"},{"t":"add","s":""},{"t":"add","s":""},{"t":"add","s":""},{"t":"gap","s":"… 167 more lines"}]},{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","date":"2024-11","precision":"month","source":"archive","provisional":false,"between":["2024-10-01","2024-11-01"],"summary":"Now blocked: Applebot-Extended, Meta-ExternalAgent and Meta-ExternalFetcher.","dir":"up","up":["Applebot-Extended","Meta-ExternalAgent","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241001000953/https://www.wsj.com/robots.txt","https://web.archive.org/web/20241101004723/https://www.wsj.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cb_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_ai_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_debt_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cyber_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_vc_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_pe_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml"},{"t":"add","s":"Sitemap: https://www.wsj.com/buyside/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"# For Buyside Search Results"},{"t":"del","s":"Disallow: /*?*term=德*"},{"t":"del","s":"Disallow: /*?*term=印*"},{"t":"del","s":"Disallow: /*?*term=办*"},{"t":"del","s":"Disallow: /*?*term=西*"},{"t":"del","s":"Disallow: /*?*term=刚*"},{"t":"del","s":"Disallow: /*?*term=网*"},{"t":"del","s":"Disallow: /*?*term=焦*"},{"t":"del","s":"Disallow: /*?*term=澳*"},{"t":"del","s":"Disallow: /*?*term=制*"},{"t":"del","s":"Disallow: /*?*term=仿*"},{"t":"del","s":"Disallow: /*?*term=护*"},{"t":"del","s":"Disallow: /*?*term=雅*"},{"t":"del","s":"Disallow: /*?*term=入*"},{"t":"del","s":"Disallow: /*?*term=订*"},{"t":"del","s":"Disallow: /*?*term=假*"},{"t":"del","s":"Disallow: /*?*term=的*"},{"t":"del","s":"Disallow: /*?*term=一*"},{"t":"del","s":"Disallow: /*?*term=是*"},{"t":"del","s":"Disallow: /*?*term=不*"},{"t":"del","s":"Disallow: /*?*term=了*"},{"t":"del","s":"Disallow: /*?*term=人*"},{"t":"del","s":"Disallow: /*?*term=我*"},{"t":"del","s":"Disallow: /*?*term=在*"},{"t":"del","s":"Disallow: /*?*term=有*"},{"t":"del","s":"Disallow: /*?*term=他*"},{"t":"del","s":"Disallow: /*?*term=这*"},{"t":"del","s":"Disallow: /*?*term=为*"},{"t":"del","s":"Disallow: /*?*term=之*"},{"t":"del","s":"Disallow: /*?*term=大*"},{"t":"del","s":"Disallow: /*?*term=来*"},{"t":"del","s":"Disallow: /*?*term=以*"},{"t":"del","s":"Disallow: /*?*term=个*"},{"t":"del","s":"Disallow: /*?*term=中*"},{"t":"del","s":"Disallow: /*?*term=上*"},{"t":"del","s":"Disallow: /*?*term=们*"},{"t":"del","s":"Disallow: /*?*term=到*"},{"t":"del","s":"Disallow: /*?*term=说*"},{"t":"del","s":"Disallow: /*?*term=国*"},{"t":"del","s":"Disallow: /*?*term=和*"},{"t":"del","s":"Disallow: /*?*term=地*"},{"t":"del","s":"Disallow: /*?*term=也*"},{"t":"del","s":"Disallow: /*?*term=子*"},{"t":"del","s":"Disallow: /*?*term=时*"},{"t":"del","s":"Disallow: /*?*term=道*"},{"t":"del","s":"Disallow: /*?*term=出*"},{"t":"del","s":"Disallow: /*?*term=而*"},{"t":"gap","s":"… 83 more lines"}]},{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","date":"2024-06","precision":"month","source":"archive","provisional":false,"between":["2024-05-01","2024-06-01"],"summary":"Now blocked: ClaudeBot, Bytespider and PerplexityBot. Now allowed: GPTBot and ChatGPT-User.","dir":"mixed","up":["ClaudeBot","Bytespider","PerplexityBot"],"down":["GPTBot","ChatGPT-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240501011459/https://www.wsj.com/robots.txt","https://web.archive.org/web/20240601000158/https://www.wsj.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml"},{"t":"del","s":"Sitemap: https://www.wsj.com/coupons/sitemap.xml"},{"t":"ctx","s":"Sitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: NewsNow"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: news-please"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Perplexity-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","date":"2023-12","precision":"month","source":"archive","provisional":false,"between":["2023-11-01","2023-12-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231101003040/https://www.wsj.com/robots.txt","https://web.archive.org/web/20231201001359/https://www.wsj.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /user/fpd/api/*"},{"t":"add","s":"Disallow: /Date(*"},{"t":"add","s":"Disallow: /auth/sso/proxy-login*"},{"t":"add","s":"Disallow: /client/"},{"t":"ctx","s":"# For Buyside Search Results"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"wsj.com","name":"The Wall Street Journal","category":"news","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-09-01","2023-10-01"],"summary":"Now blocked: GPTBot, CCBot, anthropic-ai and ChatGPT-User.","dir":"up","up":["GPTBot","CCBot","anthropic-ai","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230901000033/https://www.wsj.com/robots.txt","https://web.archive.org/web/20231001004238/https://www.wsj.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /user/fpd/api/*"},{"t":"add","s":"# For Buyside Search Results"},{"t":"add","s":"Disallow: /*?*term=德*"},{"t":"add","s":"Disallow: /*?*term=印*"},{"t":"add","s":"Disallow: /*?*term=办*"},{"t":"add","s":"Disallow: /*?*term=西*"},{"t":"add","s":"Disallow: /*?*term=刚*"},{"t":"add","s":"Disallow: /*?*term=网*"},{"t":"add","s":"Disallow: /*?*term=焦*"},{"t":"add","s":"Disallow: /*?*term=澳*"},{"t":"add","s":"Disallow: /*?*term=制*"},{"t":"add","s":"Disallow: /*?*term=仿*"},{"t":"add","s":"Disallow: /*?*term=护*"},{"t":"add","s":"Disallow: /*?*term=雅*"},{"t":"add","s":"Disallow: /*?*term=入*"},{"t":"add","s":"Disallow: /*?*term=订*"},{"t":"add","s":"Disallow: /*?*term=假*"},{"t":"add","s":"Disallow: /*?*term=的*"},{"t":"add","s":"Disallow: /*?*term=一*"},{"t":"add","s":"Disallow: /*?*term=是*"},{"t":"add","s":"Disallow: /*?*term=不*"},{"t":"add","s":"Disallow: /*?*term=了*"},{"t":"add","s":"Disallow: /*?*term=人*"},{"t":"add","s":"Disallow: /*?*term=我*"},{"t":"add","s":"Disallow: /*?*term=在*"},{"t":"add","s":"Disallow: /*?*term=有*"},{"t":"add","s":"Disallow: /*?*term=他*"},{"t":"add","s":"Disallow: /*?*term=这*"},{"t":"add","s":"Disallow: /*?*term=为*"},{"t":"add","s":"Disallow: /*?*term=之*"},{"t":"add","s":"Disallow: /*?*term=大*"},{"t":"add","s":"Disallow: /*?*term=来*"},{"t":"add","s":"Disallow: /*?*term=以*"},{"t":"add","s":"Disallow: /*?*term=个*"},{"t":"add","s":"Disallow: /*?*term=中*"},{"t":"add","s":"Disallow: /*?*term=上*"},{"t":"add","s":"Disallow: /*?*term=们*"},{"t":"add","s":"Disallow: /*?*term=到*"},{"t":"add","s":"Disallow: /*?*term=说*"},{"t":"add","s":"Disallow: /*?*term=国*"},{"t":"add","s":"Disallow: /*?*term=和*"},{"t":"add","s":"Disallow: /*?*term=地*"},{"t":"add","s":"Disallow: /*?*term=也*"},{"t":"add","s":"Disallow: /*?*term=子*"},{"t":"add","s":"Disallow: /*?*term=时*"},{"t":"add","s":"Disallow: /*?*term=道*"},{"t":"add","s":"Disallow: /*?*term=出*"},{"t":"add","s":"Disallow: /*?*term=而*"},{"t":"add","s":"Disallow: /*?*term=要*"},{"t":"add","s":"Disallow: /*?*term=于*"},{"t":"add","s":"Disallow: /*?*term=就*"},{"t":"add","s":"Disallow: /*?*term=下*"},{"t":"add","s":"Disallow: /*?*term=得*"},{"t":"add","s":"Disallow: /*?*term=可*"},{"t":"add","s":"Disallow: /*?*term=你*"},{"t":"add","s":"Disallow: /*?*term=年*"},{"t":"add","s":"Disallow: /*?*term=生*"},{"t":"add","s":"Disallow: /*?*term=自*"},{"t":"add","s":"Disallow: /*?*term=会*"},{"t":"add","s":"Disallow: /*?*term=那*"},{"t":"gap","s":"… 90 more lines"}]}]}