What is Claude-SearchBot?
Claude-SearchBot is an Anthropic crawler that navigates the web to improve search result quality in Claude. Blocking it in robots.txt prevents Anthropic from indexing the content for search, which according to Anthropic may reduce the site’s visibility in user search results.
Anthropic separates training from search. ClaudeBot collects content for model training, Claude-SearchBot works for the search feature. OpenAI follows the same pattern with GPTBot and OAI-SearchBot. For site owners this means you can opt out of training and still appear in answers that use web search.
Claude-SearchBot has its own robots.txt token. A rule for ClaudeBot does not automatically apply to it, while a blanket block for all bots does. Those blanket blocks often come from templates written before training and search were split.
Anthropic’s documentation only describes what a block does, not how much allowing the crawler increases mentions. An allowed page is not automatically cited. It only becomes readable in the first place.
What it means for your website
Open your robots.txt and look for a Claude-SearchBot group and for a User-agent: * group that disallows everything. If you want to appear in Claude with web search, neither may forbid access. The deeploupe GPTBot check does not test this token; it tests GPTBot and ClaudeBot live and reads a few more tokens from robots.txt. For Claude-SearchBot, look at the file yourself.
Related terms
More on deeploupe
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card