Skip to main content

abcnews.go.com blocks GPTBot

Browser: HTTP 200 · control fetch

AI crawlerrobots.txtLive HTTPVerdict
GPTBotOpenAI (ChatGPT)Disallow: /HTTP 200 Blocked
ClaudeBotAnthropic (Claude)—HTTP 200 Accessible
PerplexityBotPerplexity—— Accessible
Google-ExtendedGoogle (Gemini/Vertex)Disallow: /— Blocked
CCBotCommon Crawl (training data)Disallow: /— Blocked
anthropic-aiAnthropic (legacy token)Disallow: /— Blocked

robots.txt found

What the check found on abcnews.go.com

Measured on 16 November 2026, 05:45 UTC: robots.txt of abcnews.go.com read, homepage requested with the GPTBot and ClaudeBot user agent, and with a browser as control (HTTP 200).

  • GPTBot: robots.txt blocks GPTBot with “Disallow: /” in a group of its own, “User-agent: GPTBot”. Yet the server still serves it the homepage with HTTP 200, so the block holds only as long as the crawler obeys robots.txt.Collects content that may be used to train OpenAI’s models.
  • ClaudeBot: ClaudeBot got HTTP 200 in the live request; robots.txt does not name it.
  • Google-Extended: robots.txt blocks Google-Extended with “Disallow: /” in a group of its own, “User-agent: Google-Extended”. The check does not run a live request for this crawler.Not a separate crawler but a robots.txt token: controls whether content is used for Gemini training and grounding.
  • CCBot: robots.txt blocks CCBot with “Disallow: /” in a group of its own, “User-agent: CCBot”. The check does not run a live request for this crawler.Builds Common Crawl’s open web archive, which is freely accessible and is also used for AI training.
  • anthropic-ai: robots.txt blocks anthropic-ai with “Disallow: /” in a group of its own, “User-agent: anthropic-ai”. The check does not run a live request for this crawler.Older token that some robots.txt files still contain. It is no longer listed in Anthropic’s current crawler docs.

No rule of their own in robots.txt and no live request: PerplexityBot.

Excerpt from the robots.txt of abcnews.go.com

The groups that apply to one of the checked AI crawlers, as the check read them.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: *
Disallow: /search?searchtext=*
Disallow: /disneyid/*
Disallow: /assets/static/ads/*
Disallow: /cgi
Disallow: /xls
Disallow: /imp
Disallow: /kmail
Disallow: /map
Disallow: /log
Disallow: /gif
Disallow: /panel
Disallow: /0/
# + 38 more rules

Verify it yourself

curl -A "GPTBot/1.2" -s -o /dev/null -w "%{http_code}\n" https://abcnews.go.com

Keep this result?

You have seen everything. If you want, we will hold on to it — pick either or both.

By clicking the button you consent to Speakz Media GmbH storing your e-mail address in order to send you this check result once and to re-check this page in 30 days and tell you what has changed. This is not a newsletter: we will not send you advertising, and we will not use your address for anything else. You can withdraw your consent at any time — every e-mail has an unsubscribe link at the bottom. After that we delete your address. Privacy policy

GPTBot can’t load abcnews.go.com. OpenAI uses GPTBot for training data; ChatGPT search relies on a separate crawler, OAI-SearchBot, which this check does not test. If the block sits in the hosting layer (CDN, WAF, site builder) and filters bots by user agent, it may hit that crawler as well. Most blocks like this are not a deliberate choice, so they go unnoticed. The fix is usually a single allow-rule; the harder question is whether you’re then actually cited.

Related check

Is abcnews.go.com actually cited in ChatGPT?

Crawlability is only step one — a reachable site still has to be named in the AI answer to win the customer.