Is GPTBot blocked on my website?
Two things block GPTBot: a robots.txt rule such as “User-agent: GPTBot” with “Disallow: /”, or a server, CDN or firewall that answers the GPTBot user agent with an error while browsers get the page. The second is easy to miss. In our July 2026 sample, 44 of 346 reachable small-business websites did exactly that. Enter your domain and we test both live, with the HTTP status as proof.
Free, no signup. GPTBot controls whether OpenAI may use your pages for training. Whether you appear in ChatGPT search is controlled separately by OAI-SearchBot, see the table below.
Pure HTTP: we read your robots.txt per bot and fetch your homepage with the real GPTBot and ClaudeBot user-agents (control: a normal browser). No AI, no cost, no signup.
How it works
1. Enter your domain
We fetch your robots.txt and read the rules for each AI crawler by name.
2. We test the real crawler
We request your homepage with the actual GPTBot & ClaudeBot user-agent and compare it to a normal browser.
3. See the HTTP proof
Blocked or accessible — with the exact status code. A 403 to the bot but 200 to a browser is the silent hosting/WAF block.
Which AI crawlers exist and what blocking them does
Each row summarizes the operator’s own documentation (checked 16 September 2026, linked on the token). The last column says honestly which ones our free check tests.
| Token | Operator | Used for | If blocked | Our check |
|---|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI’s models. | Your content should not be used for training. Appearing in ChatGPT search is controlled by a different crawler. | robots.txt + live test |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT’s search features. | According to OpenAI the site is not shown in ChatGPT search answers, though it can still appear as a navigational link. | not checked |
| ChatGPT-User | OpenAI | Fetches a page when a person asks for it in ChatGPT. Does not crawl on its own. | According to OpenAI, robots.txt rules may not apply here because a person triggers the fetch. | not checked |
| ClaudeBot | Anthropic | Collects content that could contribute to training Anthropic’s models. | Future content of the site should be excluded from Anthropic’s training data. | robots.txt + live test |
| Claude-SearchBot | Anthropic | Reads content to improve search results in Claude. | According to Anthropic the site is not indexed for search, which may reduce its visibility in Claude search answers. | not checked |
| Claude-User | Anthropic | Fetches a page when a person asks Claude a question. | Claude cannot retrieve the page for such user requests. | not checked |
| anthropic-ai | Anthropic | Older token that some robots.txt files still contain. It is no longer listed in Anthropic’s current crawler docs. | The effect is not described in the current docs. To control Anthropic, use the three tokens above. | robots.txt only |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. According to Perplexity not used for model training. | Perplexity recommends allowing the bot so the site appears in search results. | robots.txt only |
| Perplexity-User | Perplexity | Fetches a page when a person asks Perplexity and links it in the answer. | Controls which sites such user requests can access. | not checked |
| Google-Extended | Not a separate crawler but a robots.txt token: controls whether content is used for Gemini training and grounding. | According to Google this does not affect inclusion in Google Search. | robots.txt only | |
| CCBot | Common Crawl | Builds Common Crawl’s open web archive, which is freely accessible and is also used for AI training. | The site is no longer read into future Common Crawl archives. | robots.txt only |
Recently checked domains
Live GPTBot/ClaudeBot crawlability checks people already ran — see which sites let AI crawlers in.
FAQ
- Does blocking GPTBot keep my website out of ChatGPT?
- Not by itself, according to OpenAI. GPTBot collects content that may be used for training, OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI states that each setting is independent of the other. A block in your server, CDN or firewall is different: it hits every user agent the rule matches. Our check tests GPTBot and ClaudeBot live, not OAI-SearchBot.
- How do I check if GPTBot is blocked?
- In two steps. Open yourdomain.com/robots.txt and look for a group “User-agent: GPTBot” or “User-agent: *” with “Disallow: /”. Then request your homepage with the GPTBot user agent and compare the HTTP status with a normal browser request. An error for the bot and 200 for the browser points to a bot rule in your CDN, firewall or hosting. The free check above runs both steps.
- Does a block mean I chose to block ChatGPT?
- Usually not. Our data across hundreds of small-business sites found only ~3.7% block AI crawlers on purpose via robots.txt, yet ~12.7% silently return a 403 to GPTBot while serving browsers.
- How do I unblock GPTBot?
- If robots.txt blocks it, remove or narrow the “Disallow” rule in the GPTBot group. If the server blocks it, look for bot protection in your CDN, firewall or hosting settings and allow the verified OpenAI crawlers; OpenAI publishes their IP ranges. Whether GPTBot may use your content for training is your decision. For ChatGPT search, OAI-SearchBot is the crawler to allow.
- Which crawlers do you test?
- robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and anthropic-ai, and a live HTTP test for GPTBot and ClaudeBot. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and Perplexity-User are not tested; the table shows what each of them does.
- Do I need to sign up?
- No. The check needs no account and no credit card.