mirror.co.uk blocks GPTBot
Browser: HTTP 200 · control fetch
| AI crawler | robots.txt | Live HTTP | Verdict |
|---|---|---|---|
| GPTBotOpenAI (ChatGPT) | Disallow: / | HTTP 402 | Blocked |
| ClaudeBotAnthropic (Claude) | Disallow: / | HTTP 402 | Blocked |
| PerplexityBotPerplexity | Disallow: / | — | Blocked |
| Google-ExtendedGoogle (Gemini/Vertex) | — | — | Accessible |
| CCBotCommon Crawl (Trainingsdaten) | Disallow: / | — | Blocked |
| anthropic-aiAnthropic (Legacy-Token) | Disallow: / | — | Blocked |
robots.txt found
Verify it yourself
curl -A "GPTBot/1.2" -s -o /dev/null -w "%{http_code}\n" https://mirror.co.ukGPTBot can’t load mirror.co.uk. OpenAI uses GPTBot for training data; ChatGPT search relies on a separate crawler, OAI-SearchBot, which this check does not test. If the block sits in the hosting layer (CDN, WAF, site builder) and filters bots by user agent, it may hit that crawler as well. Most blocks like this are not a deliberate choice, so they go unnoticed. The fix is usually a single allow-rule; the harder question is whether you’re then actually cited.
Related check
Is mirror.co.uk actually cited in ChatGPT?Crawlability is only step one — a reachable site still has to be named in the AI answer to win the customer.