What is a WAF or bot protection block?
A WAF or bot protection block happens when a web application firewall, a CDN or a host’s bot protection rejects crawler requests before they reach the website. robots.txt may allow everything; the crawler still only receives an error page.
These blocks live in no file the site owner maintains. They come from hosting defaults, from a CDN’s security rules or from switches such as Cloudflare’s bot protection. Many site owners therefore do not know that their site is closed to AI crawlers.
The pattern is easy to measure. A browser gets the page, the crawler gets an error code, such as 403. In our measurement across 346 reachable websites of German small businesses, exactly this happened for the GPTBot user agent on 12.7 percent of sites, while only 3.7 percent of 327 readable robots.txt files blocked an AI crawler by name.
Crawler operators know the problem too. Perplexity’s documentation notes that site owners using a web application firewall may need to explicitly allow its bots.
What it means for your website
Test your homepage with the real crawler user agent and compare it with a browser. If the status codes differ, the block sits in the firewall, CDN or hosting, not in robots.txt. The deeploupe GPTBot check shows this comparison for GPTBot and ClaudeBot. The fix is usually an exception rule in the bot protection, documented by your host or CDN.
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card