Skip to main content
deSign inFree scan

thetimes.com blocks an AI crawler

Browser: HTTP 200 · control fetch

AI crawlerrobots.txtLive HTTPVerdict
GPTBotOpenAI (ChatGPT)rule presentHTTP 200 Accessible
ClaudeBotAnthropic (Claude)Disallow: /HTTP 200 Blocked
PerplexityBotPerplexityDisallow: /— Blocked
Google-ExtendedGoogle (Gemini/Vertex)Disallow: /— Blocked
CCBotCommon Crawl (training data)Disallow: /— Blocked
anthropic-aiAnthropic (legacy token)Disallow: /— Blocked

robots.txt found

Verify it yourself

curl -A "GPTBot/1.2" -s -o /dev/null -w "%{http_code}\n" https://thetimes.com

Keep this result?

You have seen everything. If you want, we will hold on to it — pick either or both.

By clicking the button you consent to Speakz Media GmbH storing your e-mail address in order to send you this check result once and to re-check this page in 30 days and tell you what has changed. This is not a newsletter: we will not send you advertising, and we will not use your address for anything else. You can withdraw your consent at any time — every e-mail has an unsubscribe link at the bottom, one click is enough. After that we delete your address. Privacy policy

GPTBot can’t load thetimes.com. OpenAI uses GPTBot for training data; ChatGPT search relies on a separate crawler, OAI-SearchBot, which this check does not test. If the block sits in the hosting layer (CDN, WAF, site builder) and filters bots by user agent, it may hit that crawler as well. Most blocks like this are not a deliberate choice, so they go unnoticed. The fix is usually a single allow-rule; the harder question is whether you’re then actually cited.

Related check

Is thetimes.com actually cited in ChatGPT?

Crawlability is only step one — a reachable site still has to be named in the AI answer to win the customer.

Now check: are you actually cited in ChatGPT?See the data: how often small businesses silently block GPTBot