WebMD

webmd.com

Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 3 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot and CCBot. Google-Extended, Applebot-Extended, Meta-ExternalAgent, OAI-SearchBot and 5 more can reach most of it.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

3 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Allowed

0 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI	Blocked	Named, line 40	Says yes
ClaudeBotAnthropic	Blocked	Named, line 34	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
CCBotCommon Crawl	Blocked	Named, line 31	Says yes
Meta-ExternalAgentMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
BytespiderByteDance	Allowed	Not named; the catch-all rule applies	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Allowed	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Allowed	Not named; the catch-all rule applies	Unknown
Bots for AI search · Allowed
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
PerplexityBotPerplexity	Allowed	Not named; the catch-all rule applies	Suggests yes
AmazonbotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
Amzn-SearchBotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Allowed	Not named; the catch-all rule applies	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI	Allowed	Not named; the catch-all rule applies	Says not always
Claude-UserAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Perplexity-UserPerplexity	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Allowed	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 41 lines · 6 lines about AI botsSource

# Robots.txt file WebMD# Updated 8/31/2026 (CONSFE-458) User-agent: *Disallow: */search/search_results/Disallow: /500Disallow: /aim/Disallow: /api/Disallow: /bipolar-disorder/video/bipolar-fatherhoodDisallow: /bipolar-disorder/video/bipolar-toolboxDisallow: /breast-cancer/bc-treatment-21/provider-selectDisallow: /click*Disallow: /dnaDisallow: /drugs/2/search*Disallow: /drugs/ReportAbuse.aspx*Disallow: /kapi/Disallow: /mm/Disallow: /my-libraryDisallow: /news/header.phpDisallow: /pill-identification/search-results*Disallow: /search/2/api/Disallow: /Share.aspx*Disallow: /sponsored/Disallow: /static/Disallow: /story/Disallow: /webmd_static_vue/ User-agent: AdsBot-GoogleDisallow: /mm/ User-agent: CCBotDisallow: / User-agent: ClaudeBotDisallow: / User-agent: DoximityDisallow: / User-agent: GPTBotDisallow: /

Everything else we checked

Front door	Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.webmd.com/ as WalledWebBot, from a residential connection in the United States
Anti-bot software	Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	No schema.org structured data on the homepage. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result.
Terms of Service	www.webmd.com/about-webmd-policies/about-terms-and-conditions-of-use (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context. … you have a good faith belief that the complained of use of the materials is not authorized by the copyright owner, its agent, or the law.matched "agent" · reads like a restriction Harvesting, scraping or otherwise collecting information about others, including email addresses, without their consent for posting or viewing comments;matched "scraping" You agree to defend, indemnify, and hold WebMD, its officers, directors, employees, agents, licensors, and suppliers, harmless from and against any claims, actions or demands, liabilities and settlements, including, without limitation, reasonable legal and accounting fees, resulting from, or alleged…matched "agents"
Not found	No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags.

History

45 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Sep 2026
Less blocking Now allowed: ChatGPT-User.
Found in the Internet Archive's copies: changed between 5 Aug 2026 and 3 Sep 2026. Evidence: before · after

Show exactly what changed
# Robots.txt file WebMD
- # Updated 1/8/2025
+ # Updated 8/31/2026 (CONSFE-458)

…
Disallow: */search/search_results/
- Disallow: /drugs/ReportAbuse.aspx*
- Disallow: */cm/
- Disallow: /Share.aspx*
- Disallow: /mm/
- Disallow: /kapi/
- Disallow: /click*
- Disallow: /drugs/2/search*
- Disallow: /sponsored/
- Disallow: /api/
- Disallow: /search/2/api/
- Disallow: /webmd_static_vue/
+ Disallow: /500
Disallow: /aim/
- Disallow: /6-tools-for-managing-your-medications
- Disallow: /attr-cm-warning-signs
- Disallow: /bipolar-disorder/bipolar-depression-management
+ Disallow: /api/
Disallow: /bipolar-disorder/video/bipolar-fatherhood
Disallow: /bipolar-disorder/video/bipolar-toolbox
- Disallow: /bluechew-review
- Disallow: /brain/treating-gmg-video
- Disallow: /brain/video/treating-gmg-video
Disallow: /breast-cancer/bc-treatment-21/provider-select
- Disallow: /cbd/about-cbd
- Disallow: /cbd/best-cbd-gummies-for-anxiety
- Disallow: /cbd/best-cbd-oil-for-anxiety
- Disallow: /cbd/best-cbd-oil-for-pain
- Disallow: /cbd/best-cbd-oil-of-2023
- Disallow: /cbd/best-full-spectrum-cbd-oils
- Disallow: /cbd/best-gummies-of-2023
- Disallow: /cbd/cbd-101-concentrated-cannabidiol
- Disallow: /cbd/cbd-for-dogs
- Disallow: /cbd/cbd-for-sleep
- Disallow: /cbd/cbdfx-review
- Disallow: /cbd/cbdistillery-review
- Disallow: /cbd/koi-cbd-review
- Disallow: /chemo-combo
- Disallow: /cholesterol-management/high-cholesterol-treatment
- Disallow: /choose-cancer-care
- Disallow: /diagnosed-with-glioblastoma
+ Disallow: /click*
Disallow: /dna
- Disallow: /dna/meet-your-match
- Disallow: /dna/meet-your-match-acne-prone-skin-routine
- Disallow: /dna/meet-your-match-combination-to-oily-skin-routine
- Disallow: /dna/meet-your-match-normal-to-dry-skin-routine
- Disallow: /dupuytrens/default.htm
- Disallow: /dupuytrens/treatment
- Disallow: /dupuytrens/what-to-expect
- Disallow: /ear-health-support/balance
- Disallow: /ear-health-support/daily-care
- Disallow: /ear-health-support/default.htm
- Disallow: /ear-health-support/hearing-loss
- Disallow: /epilepsy-seizures-rapid-rescue-single-treatment
… 63 more lines
Feb 2025
More blocking Now blocked: ChatGPT-User.
Found in the Internet Archive's copies: changed between 1 Jan 2025 and 1 Feb 2025. Evidence: before · after

Show exactly what changed
# Robots.txt file WebMD
- # Updated 11/8/2024
+ # Updated 1/8/2025

…

- Sitemap: https://www.webmd.com/rx/sitemap.xml
- Sitemap: https://www.webmd.com/sitemapxml/sitemap.xml
- Sitemap: https://www.webmd.com/sitemapxml/news.xml
+ User-agent: AdsBot-Google
+ Disallow: /mm/
+
+ User-agent: ChatGPT-User
+ Disallow: /
Dec 2024
More blocking Now blocked: ClaudeBot and CCBot.
Found in the Internet Archive's copies: changed between 1 Nov 2024 and 1 Dec 2024. Evidence: before · after

Show exactly what changed
# Robots.txt file WebMD
- # Updated 8/24/2023
+ # Updated 11/8/2024

…
Disallow: /aim/
+ Disallow: /6-tools-for-managing-your-medications
+ Disallow: /attr-cm-warning-signs
+ Disallow: /bipolar-disorder/bipolar-depression-management
+ Disallow: /bipolar-disorder/video/bipolar-fatherhood
+ Disallow: /bipolar-disorder/video/bipolar-toolbox
+ Disallow: /bluechew-review
+ Disallow: /brain/treating-gmg-video
+ Disallow: /brain/video/treating-gmg-video
+ Disallow: /breast-cancer/bc-treatment-21/provider-select
+ Disallow: /cbd/about-cbd
+ Disallow: /cbd/best-cbd-gummies-for-anxiety
+ Disallow: /cbd/best-cbd-oil-for-anxiety
+ Disallow: /cbd/best-cbd-oil-for-pain
+ Disallow: /cbd/best-cbd-oil-of-2023
+ Disallow: /cbd/best-full-spectrum-cbd-oils
+ Disallow: /cbd/best-gummies-of-2023
+ Disallow: /cbd/cbd-101-concentrated-cannabidiol
+ Disallow: /cbd/cbd-for-dogs
+ Disallow: /cbd/cbd-for-sleep
+ Disallow: /cbd/cbdfx-review
+ Disallow: /cbd/cbdistillery-review
+ Disallow: /cbd/koi-cbd-review
+ Disallow: /chemo-combo
+ Disallow: /cholesterol-management/high-cholesterol-treatment
+ Disallow: /choose-cancer-care
+ Disallow: /diagnosed-with-glioblastoma
+ Disallow: /dna
+ Disallow: /dna/meet-your-match
+ Disallow: /dna/meet-your-match-acne-prone-skin-routine
+ Disallow: /dna/meet-your-match-combination-to-oily-skin-routine
+ Disallow: /dna/meet-your-match-normal-to-dry-skin-routine
+ Disallow: /dupuytrens/default.htm
+ Disallow: /dupuytrens/treatment
+ Disallow: /dupuytrens/what-to-expect
+ Disallow: /ear-health-support/balance
+ Disallow: /ear-health-support/daily-care
+ Disallow: /ear-health-support/default.htm
+ Disallow: /ear-health-support/hearing-loss
+ Disallow: /epilepsy-seizures-rapid-rescue-single-treatment
+ Disallow: /friends-through-gvhd
+ Disallow: /gum-disease-treatment
+ Disallow: /hair-transplant-services
+ Disallow: /hims-hard-mints
+ Disallow: /how-caregivers-helped-loved-ones-with-attr-cm
+ Disallow: /knee-osteoarthritis-clinical-trial
+ Disallow: /lung/copd/treat-copd-21/copd-new-treatment
+ Disallow: /mental-health-genetic-testing
+ Disallow: /mother-to-baby-observational-research-studies
+ Disallow: /neuromyelitis-optica-spectrum-disorder-nmosd-treatment
+ Disallow: /nvaf-stroke-risk-treatment
+ Disallow: /oral-and-maxillofacial-surgeon
+ Disallow: /parenting/cpp-21/what-is-cpp
+ Disallow: /peyronies-treatment-option/talk-to-doctor
+ Disallow: /peyronies-treatment-option/treatment-option
… 33 more lines
Sep 2023
More blocking Now blocked: GPTBot.
Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after

Show exactly what changed
- # Robots.txt file WebMD
- # Updated: Oct 2022
+ # Robots.txt file WebMD
+ # Updated 8/24/2023

…
Disallow: /Share.aspx*
+ Disallow: /mm/
+ Disallow: /kapi/
+ Disallow: /click*
+ Disallow: /drugs/2/search*
+ Disallow: /sponsored/
+ Disallow: /api/
+ Disallow: /search/2/api/
+ Disallow: /webmd_static_vue/
+ Disallow: /aim/

+ User-agent: GPTBot
+ Disallow: /
+

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.