Mount Sinai
mountsinai.orgAllows AI
The site has no robots.txt (the public rules file for bots): asking for one returned an HTML page instead of a robots.txt file. No rules means no bot is restricted.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Allowed
0 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Allowed | |||
| GPTBotOpenAI | Allowed | No rules apply | Says yes |
| ClaudeBotAnthropic | Allowed | No rules apply | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | No rules apply | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | No rules apply | Says yes |
| CCBotCommon Crawl | Allowed | No rules apply | Says yes |
| Meta-ExternalAgentMeta | Allowed | No rules apply | Suggests yes |
| BytespiderByteDance | Allowed | No rules apply | No statement found |
| MistralAI-TrainingMistral AI | Allowed | No rules apply | Says yes |
| AI2BotAllen Institute for AI | Allowed | No rules apply | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | No rules apply | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | No rules apply | Says yes |
| Claude-SearchBotAnthropic | Allowed | No rules apply | Says yes |
| PerplexityBotPerplexity | Allowed | No rules apply | Suggests yes |
| AmazonbotAmazon | Allowed | No rules apply | Says yes |
| Amzn-SearchBotAmazon | Allowed | No rules apply | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | No rules apply | Says yes |
| MistralAI-IndexMistral AI | Allowed | No rules apply | Suggests yes |
| Meta-WebIndexerMeta | Allowed | No rules apply | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | No rules apply | Says not always |
| Claude-UserAnthropic | Allowed | No rules apply | Says yes |
| Perplexity-UserPerplexity | Allowed | No rules apply | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | No rules apply | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | No rules apply | Says not always |
| Amzn-UserAmazon | Allowed | No rules apply | Says not always |
| MistralAI-UserMistral AI | Allowed | No rules apply | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | No rules apply | |
| BingbotRegular search engine, for comparison | Allowed | No rules apply | |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
This site has no robots.txt: asking for one returned an HTML page instead of a robots.txt file. No rules file means no bot is restricted.
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.mountsinai.org/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Anti-bot software | Served through Akamai.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: WebSite, SearchAction, MedicalOrganization, PostalAddress, ContactPoint. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | www.mountsinai.org/tou (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context.
|
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags. |
History
44 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Oct 2026Signals Stopped naming CCBot; access unchanged.Changed between the Internet Archive's copy from 29 Sep 2026 and our own check on 19 Oct 2026. Not confirmed yet: seen once so far. Evidence: before · after
Show exactly what changed
- User-agent: *- Disallow: /cgi-bin/- Disallow: /tmp/- Disallow: /vgn-ext/templating- Disallow: /*lp/- Disallow: /insurance/- Disallow: /about/preparedness/coronavirus/staff-resources*- Disallow: /about/covid19/staff-resources*- User-agent: Alexabot- Disallow: /about-us/newsroom/- Crawl-delay: 5-- User-agent: CCBot- Disallow:- Crawl-delay: 20-- User-agent: BLEXBot- Disallow:- Crawl-delay: 10-- User-agent: Findxbot- Disallow:- Crawl-delay: 10-- User-agent: rogerbot- Disallow: /about-us/newsroom/- Crawl-delay: 10-- User-agent: Yeti- Disallow:- Crawl-delay: 10-- User-agent: AhrefsBot- Disallow:- Crawl-delay: 10-- User-agent: SEMrushBot- Disallow:- Crawl-delay: 10-- User-agent: SEMrushBot-SA- Disallow:- Crawl-delay: 10-- User-agent: SEMrushBot-Desktop- Disallow:- Crawl-delay: 10-- User-agent: opensiteexplorer.org- Disallow:- Crawl-delay: 10-- User-agent: dotbot- Disallow:- Crawl-delay: 20-- User-agent: Slurp- Disallow:- Crawl-delay: 10… 28 more lines
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.