The Wall Street Journal

wsj.com

Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 21 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot and ChatGPT-User can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

8 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

7 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Some blocked

6 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI	Allowed	Named, line 16	Says yes
ClaudeBotAnthropic	Blocked	Not named; the catch-all rule applies	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Blocked	Not named; the catch-all rule applies	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Blocked	Not named; the catch-all rule applies	Says yes
CCBotCommon Crawl	Blocked	Not named; the catch-all rule applies	Says yes
Meta-ExternalAgentMeta	Blocked	Not named; the catch-all rule applies	Suggests yes
BytespiderByteDance	Blocked	Not named; the catch-all rule applies	No statement found
MistralAI-TrainingMistral AI	Blocked	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Blocked	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Blocked	Not named; the catch-all rule applies	Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI	Allowed	Named, line 16	Says yes
Claude-SearchBotAnthropic	Blocked	Not named; the catch-all rule applies	Says yes
PerplexityBotPerplexity	Blocked	Not named; the catch-all rule applies	Suggests yes
AmazonbotAmazon	Blocked	Not named; the catch-all rule applies	Says yes
Amzn-SearchBotAmazon	Blocked	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Blocked	Not named; the catch-all rule applies	Says yes
MistralAI-IndexMistral AI	Blocked	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Blocked	Not named; the catch-all rule applies	Suggests yes
AI assistants · Some blocked
ChatGPT-UserOpenAI	Allowed	Named, line 16	Says not always
Claude-UserAnthropic	Blocked	Not named; the catch-all rule applies	Says yes
Perplexity-UserPerplexity	Blocked	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Blocked	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Blocked	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Blocked	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Blocked	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Named
BingbotRegular search engine, for comparison	Allowed	Named

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

GoogleOther	Has rules	line 18	Documented by its operator
google-cloudvertexbot	Has rules	line 36	Documented by its operator
Applebot	Has rules	line 38	Documented by its operator

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 159 lines · 127 lines about AI bots Source

# NOTICE: Collection of content and other data on https://www.wsj.com/ through# automated means is prohibited unless you have express written# permission from Dow Jones & Company, Inc. and may only be conducted for the# limited purpose contained in said permission.## Dow Jones & Company, Inc. Terms of Use may be found at# https://www.dowjones.com/terms-of-use/## If you would like to apply for permission to license the# intellectual property and/or other materials of Dow Jones & Company, Inc.’s# brands, please contact us via email at copyright@dowjones.com. User-agent: *Disallow: /… 14 lines not about AI bots User-agent: googlebotUser-agent: googlebot-imageUser-agent: GoogleOtherUser-agent: Googlebot-VideoUser-agent: Google-InspectionToolUser-agent: AdsBot-GoogleUser-agent: AdsBot-Google-MobileUser-agent: AdsBot-Google-Mobile-AppsUser-agent: Storebot-GoogleUser-agent: google-read-aloudUser-agent: bingbotUser-agent: msnbotUser-agent: bingpreviewUser-agent: slurpUser-agent: yahooUser-agent: baiduspiderUser-agent: PinterestbotUser-agent: YetiUser-agent: MojeekBotUser-agent: 360SpiderUser-agent: google-cloudvertexbotUser-agent: duckduckbotUser-agent: ApplebotUser-agent: flipboardUser-agent: qwantbotUser-agent: SeznamBot   User-agent: proximicUser-agent: admantxUser-agent: thetradedeskUser-agent: outbrainUser-agent: ias_crawlerUser-agent: AmazonAdBotUser-agent: pubmaticUser-agent: smartologybotUser-agent: parselybotUser-agent: BomboraBotUser-agent: IllumaBot/1.0User-agent: Publication-Access-for-FacebookUser-agent: WallieUser-agent: mediapartners-google  User-agent: Screaming Frog SEO SpiderUser-agent: AhrefsBotUser-agent: SemrushBotUser-agent: SimilarWebBotUser-agent: SISTRIXUser-agent: botifyUser-agent: Chrome-Lighthouse   User-agent: ChatGPT-UserUser-agent: GPTBotUser-agent: OAI-SearchBot  User-agent: facebookexternalhitUser-agent: facebotUser-agent: twitterbotUser-agent: linkedinbotUser-agent: snapchatUser-agent: sentryUser-agent: IframelyUser-agent: VocabtrackerUser-agent: EpvzCrawl6194680250User-agent: CitoidUser-agent: ZoteroTranslationServer   Allow: /Disallow: /article_email/*Disallow: /user/*Disallow: /pdf/documents/*Disallow: /login/*Disallow: /acct/*Disallow: /msgcenter/*Disallow: /setup/*Disallow: /marketing/*Disallow: /public/article/*Disallow: /public/resources/documents/*Disallow: /public/search/Disallow: /public/search*Disallow: /search*Disallow: /public/page/wsj-x-marketing.htmlDisallow: /public/page/news-media-marketing.htmlDisallow: /public/page/0_0_WP_RT_MARKETING.htmlDisallow: /news/articles/SB2*Disallow: /news/articles/SB3*Disallow: /news/articles/SB4*Disallow: /articles/SB2*Disallow: /articles/SB3*Disallow: /articles/SB4*Disallow: /article/AP*Disallow: /article/BT-CO*Disallow: /article/DN-CO*Disallow: /article/PR-CO*Disallow: /article/HUG*Disallow: /video/search/*Disallow: /articles/BT-CO*Disallow: /articles/DN-CO*Disallow: /articles/PR-CO*Disallow: /news/articles/BT-CO*Disallow: /news/articles/DN-CO*Disallow: /news/articles/PR-CO*Disallow: /catchup/*Disallow: /articles/the-meaning-behind-juneteenth-11592413234Disallow: /emailservice/*Disallow: /emailsignup/*Disallow: /insetsrv/v1/*Disallow: /user/fpd/api/*Disallow: /Date(*Disallow: /auth/sso/proxy-login*Disallow: /client/ # For Buyside Search ResultsDisallow: /buyside/search-results?*term=*# Don't crawl non-indexable sitesDisallow: /*?type=mdc_*&id=*Disallow: /*?id=*&type=mdc_*Disallow: /market-data/quotes/*/options/*Disallow: /subscribe/?inttrackingCode=*Disallow: /subscribe/?template=*  Sitemap: https://www.wsj.com/sitemap.xmlSitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xmlSitemap: https://www.wsj.com/wsj_video_recent.xmlSitemap: https://www.wsj.com/sitemap_topics.xmlSitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xmlSitemap: https://www.wsj.com/live_news_sitemap.xmlSitemap: https://www.wsj.com/authors_sitemap.xmlSitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xmlSitemap: https://www.wsj.com/buyside/sitemap.xmlSitemap: https://www.wsj.com/wsj_quote_index_sitemap.xmlSitemap: https://www.wsj.com/wsjsitemaps/wsj_recipes.xmlSitemap: https://www.wsj.com/sitemap_topic_collections.xml   … 16 lines not about AI bots

Everything else we checked

Front door	We did not visit: the rules ask our bot to stay out.
llms.txt	Disallowed for AgentAccessIndexBot by robots.txt
ai.txt	Disallowed for AgentAccessIndexBot by robots.txt
TDMRep	Disallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report)
Page tags	Homepage not read.
Anti-bot software	Served through Amazon CloudFront.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	Homepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of Service	No Terms of Service page found.
Not found	No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

47 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Jun 2026
More blocking Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.
Found in the Internet Archive's copies: changed between 1 May 2026 and 2 Jun 2026. Evidence: before · after

Show exactly what changed
+ # NOTICE: Collection of content and other data on https://www.wsj.com/ through
+ # automated means is prohibited unless you have express written
+ # permission from Dow Jones & Company, Inc. and may only be conducted for the
+ # limited purpose contained in said permission.
+ #
+ # Dow Jones & Company, Inc. Terms of Use may be found at
+ # https://www.dowjones.com/terms-of-use/
+ #
+ # If you would like to apply for permission to license the
+ # intellectual property and/or other materials of Dow Jones & Company, Inc.’s
+ # brands, please contact us via email at copyright@dowjones.com.
+
User-agent: *
- Sitemap: https://www.wsj.com/sitemap.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml
- Sitemap: https://www.wsj.com/sitemap_topics.xml
- Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml
- Sitemap: https://www.wsj.com/live_news_sitemap.xml
- Sitemap: https://www.wsj.com/wsj_graphics_sitemap.xml
- Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_types.xml
- Sitemap: https://www.wsj.com/authors_sitemap.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml
- Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp
- Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp
- Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
- Sitemap: https://www.wsj.com/buyside/sitemap.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml
+ Disallow: /

+ User-agent: googlebot
+ User-agent: googlebot-image
+ User-agent: GoogleOther
+ User-agent: Googlebot-Video
+ User-agent: Google-InspectionTool
+ User-agent: AdsBot-Google
+ User-agent: AdsBot-Google-Mobile
+ User-agent: AdsBot-Google-Mobile-Apps
+ User-agent: Storebot-Google
+ User-agent: google-read-aloud
+ User-agent: mediapartners-google
+ User-agent: bingbot
+ User-agent: msnbot
+ User-agent: bingpreview
+ User-agent: slurp
+ User-agent: yahoo
+ User-agent: baiduspider
+ User-agent: Pinterestbot
+ User-agent: Yeti
+ User-agent: MojeekBot
+ User-agent: 360Spider
+ User-agent: google-cloudvertexbot
+ User-agent: duckduckbot
+ User-agent: Applebot
+ User-agent: flipboard
+ User-agent: qwantbot
+ User-agent: SeznamBot
+
+
+
… 167 more lines
Nov 2024
More blocking Now blocked: Applebot-Extended, Meta-ExternalAgent and Meta-ExternalFetcher.
Found in the Internet Archive's copies: changed between 1 Oct 2024 and 1 Nov 2024. Evidence: before · after

Show exactly what changed
Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cb_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_ai_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_debt_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cyber_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_vc_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_pe_google_news.xml
- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml
- Sitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml
+ Sitemap: https://www.wsj.com/buyside/sitemap.xml
+ Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml

…
# For Buyside Search Results
- Disallow: /*?*term=德*
- Disallow: /*?*term=印*
- Disallow: /*?*term=办*
- Disallow: /*?*term=西*
- Disallow: /*?*term=刚*
- Disallow: /*?*term=网*
- Disallow: /*?*term=焦*
- Disallow: /*?*term=澳*
- Disallow: /*?*term=制*
- Disallow: /*?*term=仿*
- Disallow: /*?*term=护*
- Disallow: /*?*term=雅*
- Disallow: /*?*term=入*
- Disallow: /*?*term=订*
- Disallow: /*?*term=假*
- Disallow: /*?*term=的*
- Disallow: /*?*term=一*
- Disallow: /*?*term=是*
- Disallow: /*?*term=不*
- Disallow: /*?*term=了*
- Disallow: /*?*term=人*
- Disallow: /*?*term=我*
- Disallow: /*?*term=在*
- Disallow: /*?*term=有*
- Disallow: /*?*term=他*
- Disallow: /*?*term=这*
- Disallow: /*?*term=为*
- Disallow: /*?*term=之*
- Disallow: /*?*term=大*
- Disallow: /*?*term=来*
- Disallow: /*?*term=以*
- Disallow: /*?*term=个*
- Disallow: /*?*term=中*
- Disallow: /*?*term=上*
- Disallow: /*?*term=们*
- Disallow: /*?*term=到*
- Disallow: /*?*term=说*
- Disallow: /*?*term=国*
- Disallow: /*?*term=和*
- Disallow: /*?*term=地*
- Disallow: /*?*term=也*
- Disallow: /*?*term=子*
- Disallow: /*?*term=时*
- Disallow: /*?*term=道*
- Disallow: /*?*term=出*
- Disallow: /*?*term=而*
… 83 more lines
Jun 2024
Mixed Now blocked: ClaudeBot, Bytespider and PerplexityBot. Now allowed: GPTBot and ChatGPT-User.
Found in the Internet Archive's copies: changed between 1 May 2024 and 1 Jun 2024. Evidence: before · after

Show exactly what changed
Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml
- Sitemap: https://www.wsj.com/coupons/sitemap.xml
Sitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml
…

- User-agent: GPTBot
- Disallow: /
-
- User-agent: ChatGPT-User
- Disallow: /
-
User-agent: anthropic-ai
…

+ User-agent: Bytespider
+ Disallow: /
+
+ User-agent: ClaudeBot
+ Disallow: /
+
+ User-agent: Claude-Web
+ Disallow: /
+
+ User-agent: FacebookBot
+ Disallow: /
+
+ User-agent: magpie-crawler
+ Disallow: /
+
+ User-agent: NewsNow
+ Disallow: /
+
+ User-agent: news-please
+ Disallow: /
+
+ User-agent: PerplexityBot
+ Disallow: /
+
+ User-agent: Perplexity-ai
+ Disallow: /
+
Dec 2023
More blocking Now blocked: Google-Extended.
Found in the Internet Archive's copies: changed between 1 Nov 2023 and 1 Dec 2023. Evidence: before · after

Show exactly what changed
Disallow: /user/fpd/api/*
+ Disallow: /Date(*
+ Disallow: /auth/sso/proxy-login*
+ Disallow: /client/
# For Buyside Search Results
…

+ User-agent: Google-Extended
+ Disallow: /
+
Oct 2023
More blocking Now blocked: GPTBot, CCBot, anthropic-ai and ChatGPT-User.
Found in the Internet Archive's copies: changed between 1 Sep 2023 and 1 Oct 2023. Evidence: before · after

Show exactly what changed
Disallow: /user/fpd/api/*
+ # For Buyside Search Results
+ Disallow: /*?*term=德*
+ Disallow: /*?*term=印*
+ Disallow: /*?*term=办*
+ Disallow: /*?*term=西*
+ Disallow: /*?*term=刚*
+ Disallow: /*?*term=网*
+ Disallow: /*?*term=焦*
+ Disallow: /*?*term=澳*
+ Disallow: /*?*term=制*
+ Disallow: /*?*term=仿*
+ Disallow: /*?*term=护*
+ Disallow: /*?*term=雅*
+ Disallow: /*?*term=入*
+ Disallow: /*?*term=订*
+ Disallow: /*?*term=假*
+ Disallow: /*?*term=的*
+ Disallow: /*?*term=一*
+ Disallow: /*?*term=是*
+ Disallow: /*?*term=不*
+ Disallow: /*?*term=了*
+ Disallow: /*?*term=人*
+ Disallow: /*?*term=我*
+ Disallow: /*?*term=在*
+ Disallow: /*?*term=有*
+ Disallow: /*?*term=他*
+ Disallow: /*?*term=这*
+ Disallow: /*?*term=为*
+ Disallow: /*?*term=之*
+ Disallow: /*?*term=大*
+ Disallow: /*?*term=来*
+ Disallow: /*?*term=以*
+ Disallow: /*?*term=个*
+ Disallow: /*?*term=中*
+ Disallow: /*?*term=上*
+ Disallow: /*?*term=们*
+ Disallow: /*?*term=到*
+ Disallow: /*?*term=说*
+ Disallow: /*?*term=国*
+ Disallow: /*?*term=和*
+ Disallow: /*?*term=地*
+ Disallow: /*?*term=也*
+ Disallow: /*?*term=子*
+ Disallow: /*?*term=时*
+ Disallow: /*?*term=道*
+ Disallow: /*?*term=出*
+ Disallow: /*?*term=而*
+ Disallow: /*?*term=要*
+ Disallow: /*?*term=于*
+ Disallow: /*?*term=就*
+ Disallow: /*?*term=下*
+ Disallow: /*?*term=得*
+ Disallow: /*?*term=可*
+ Disallow: /*?*term=你*
+ Disallow: /*?*term=年*
+ Disallow: /*?*term=生*
+ Disallow: /*?*term=自*
+ Disallow: /*?*term=会*
+ Disallow: /*?*term=那*
… 90 more lines

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.