What is robots.txt?
robots.txt is a text file in a website’s root directory in which site owners specify which crawlers may fetch which areas. The format is standardized at the IETF as the Robots Exclusion Protocol. Its rules are a request to crawlers, not a technical access control.
The file consists of groups. Each group starts with one or more user-agent lines and contains allow and disallow rules for paths. A crawler follows the group that matches its name most specifically and otherwise falls back to the group with the asterisk. RFC 9309 states explicitly that these rules are not a form of access authorization.
For AI crawlers, robots.txt is the official control channel. OpenAI, Anthropic, Perplexity, Google and Apple each document their own tokens; for OpenAI, Anthropic, Google and Apple they let you control training and search separately. The file only governs what a crawler should fetch, though, not whether you get mentioned in an answer.
Deliberate blocks against AI crawlers are rarer than people think. In our measurement, 3.7 percent of 327 German small business websites with a readable robots.txt fully blocked an AI crawler by name. The same sample showed considerably more blocks at server level that appear in no robots.txt at all.
What it means for your website
Open yourdomain.com/robots.txt in a browser and read the groups for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the asterisk. A disallow line with a single slash blocks everything. The deeploupe GPTBot check reads the rules for six AI tokens and additionally tests live whether your server actually lets GPTBot and ClaudeBot through.
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card