webmd.com blocks GPTBot
Browser: HTTP 200 · control fetch
| AI crawler | robots.txt | Live HTTP | Verdict |
|---|---|---|---|
| GPTBotOpenAI (ChatGPT) | Disallow: / | HTTP 200 | Blocked |
| ClaudeBotAnthropic (Claude) | Disallow: / | HTTP 200 | Blocked |
| PerplexityBotPerplexity | — | — | Accessible |
| Google-ExtendedGoogle (Gemini/Vertex) | — | — | Accessible |
| CCBotCommon Crawl (training data) | Disallow: / | — | Blocked |
| anthropic-aiAnthropic (legacy token) | — | — | Accessible |
robots.txt found
What the check found on webmd.com
Measured on 11 November 2026, 22:15 UTC: robots.txt of webmd.com read, homepage requested with the GPTBot and ClaudeBot user agent, and with a browser as control (HTTP 200).
- GPTBot: robots.txt blocks GPTBot with “Disallow: /” in a group of its own, “User-agent: GPTBot”. Yet the server still serves it the homepage with HTTP 200, so the block holds only as long as the crawler obeys robots.txt.Collects content that may be used to train OpenAI’s models.
- ClaudeBot: robots.txt blocks ClaudeBot with “Disallow: /” in a group of its own, “User-agent: ClaudeBot”. Yet the server still serves it the homepage with HTTP 200, so the block holds only as long as the crawler obeys robots.txt.Collects content that could contribute to training Anthropic’s models.
- CCBot: robots.txt blocks CCBot with “Disallow: /” in a group of its own, “User-agent: CCBot”. The check does not run a live request for this crawler.Builds Common Crawl’s open web archive, which is freely accessible and is also used for AI training.
No rule of their own in robots.txt and no live request: PerplexityBot, Google-Extended, anthropic-ai.
Excerpt from the robots.txt of webmd.com
The groups that apply to one of the checked AI crawlers, as the check read them.
User-agent: * Disallow: */search/search_results/ Disallow: /500 Disallow: /aim/ Disallow: /api/ Disallow: /bipolar-disorder/video/bipolar-fatherhood Disallow: /bipolar-disorder/video/bipolar-toolbox Disallow: /breast-cancer/bc-treatment-21/provider-select Disallow: /click* Disallow: /dna Disallow: /drugs/2/search* Disallow: /drugs/ReportAbuse.aspx* Disallow: /kapi/ # + 10 more rules User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: GPTBot Disallow: /
Verify it yourself
curl -A "GPTBot/1.2" -s -o /dev/null -w "%{http_code}\n" https://webmd.comGPTBot can’t load webmd.com. OpenAI uses GPTBot for training data; ChatGPT search relies on a separate crawler, OAI-SearchBot, which this check does not test. If the block sits in the hosting layer (CDN, WAF, site builder) and filters bots by user agent, it may hit that crawler as well. Most blocks like this are not a deliberate choice, so they go unnoticed. The fix is usually a single allow-rule; the harder question is whether you’re then actually cited.
Related check
Is webmd.com actually cited in ChatGPT?Crawlability is only step one — a reachable site still has to be named in the AI answer to win the customer.