What is GPTBot?
GPTBot is OpenAI’s web crawler that fetches publicly accessible pages to collect content for training future models. Site owners can exclude it through the user agent token GPTBot in robots.txt, separately from OpenAI’s other crawlers.
OpenAI documents several crawlers with different jobs. GPTBot is the one for training data. According to OpenAI, disallowing GPTBot signals that a site’s content should not be used for training generative models. Search features in ChatGPT use a separate crawler, OAI-SearchBot, and user triggered fetches run through ChatGPT-User.
That is why blocking GPTBot has a different effect than often assumed: it concerns training first, not directly whether a page can appear in a current ChatGPT search answer. If you want to stop all OpenAI access, you have to set rules for each token.
In practice many blocks do not come from robots.txt at all but from bot protection at the hosting or CDN level. The crawler receives an error page while regular browsers see the site, and the owner often has no idea.
What it means for your website
Check two things separately: what your robots.txt says for GPTBot and what your server actually returns to a request with the GPTBot user agent. deeploupe’s free GPTBot check reads the robots.txt rules and fetches your homepage live with the GPTBot user agent and a normal browser for comparison. It does not test OAI-SearchBot or ChatGPT-User.
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card