What is ClaudeBot?
ClaudeBot is Anthropic’s web crawler. It collects public web content that, according to Anthropic, may contribute to training Claude models. Disallowing ClaudeBot in robots.txt signals that the site’s future content should be excluded from those training datasets.
Anthropic runs several fetchers with separate jobs. ClaudeBot is for model training. Claude-SearchBot builds a search index, and Claude-User fetches pages when a person asks Claude a question. Each has its own robots.txt token, so you can allow or block them one by one.
That split matters because blocking ClaudeBot does something different from blocking the other two. According to Anthropic, it affects training data. The documentation states that blocking Claude-SearchBot or Claude-User may reduce visibility in user-directed search results. If you only want to opt out of training, block ClaudeBot specifically and let the others through.
Accidental blocks are more common than deliberate ones. Hosting firewalls and bot protection often return an error to crawlers even when robots.txt allows everything. In our measurement across 346 reachable websites of German small businesses, 12.7 percent blocked the GPTBot user agent while serving a normal browser. ClaudeBot runs into the same pattern whenever bot protection filters by user agent.
What it means for your website
Start by looking for a ClaudeBot group in your robots.txt. Then request your homepage with the real ClaudeBot user agent and compare the response with a browser. The free deeploupe GPTBot check does exactly that: it reads the robots.txt rules and tests ClaudeBot and GPTBot live. It does not test Claude-SearchBot or Claude-User, so check those tokens in your robots.txt yourself.
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card