bestdoulas.com blocks GPTBot
Browser: HTTP 200 · control fetch
| AI crawler | robots.txt | Live HTTP | Verdict |
|---|---|---|---|
| GPTBotOpenAI (ChatGPT) | — | HTTP 429 | Blocked |
| ClaudeBotAnthropic (Claude) | — | HTTP 200 | Accessible |
| PerplexityBotPerplexity | — | — | Accessible |
| Google-ExtendedGoogle (Gemini/Vertex) | — | — | Accessible |
| CCBotCommon Crawl (Trainingsdaten) | — | — | Accessible |
| anthropic-aiAnthropic (Legacy-Token) | — | — | Accessible |
robots.txt found
Verify it yourself
curl -A "GPTBot/1.2" -s -o /dev/null -w "%{http_code}\n" https://bestdoulas.comGPTBot can’t load bestdoulas.com. OpenAI uses GPTBot for training data; ChatGPT search relies on a separate crawler, OAI-SearchBot, which this check does not test. If the block sits in the hosting layer (CDN, WAF, site builder) and filters bots by user agent, it may hit that crawler as well. Most blocks like this are not a deliberate choice, so they go unnoticed. The fix is usually a single allow-rule; the harder question is whether you’re then actually cited.
Related check
Is bestdoulas.com actually cited in ChatGPT?Crawlability is only step one — a reachable site still has to be named in the AI answer to win the customer.