Small businesses barely block ChatGPT on purpose — but their hosting does it for them
We measured 346 real local businesses (pharmacy, café, trades, medical practice, hotel) for whether they keep the AI crawlers out. The result inverts the obvious assumption.
of local businesses serve a human the page normally (HTTP 200) — and block the real GPTBot, usually with a 403. Roughly one in eight. Almost none of them decided to.
Two very different metrics
We measure them separately because they have completely different causes. Metric A is a deliberate robots.txt rule that names and blocks an AI bot. Metric B is the server itself (or its CDN/WAF/site builder) refusing the real GPTBot user-agent — usually without the owner ever knowing.
Result A: only 3.7% of businesses block *any* AI crawler on purpose via robots.txt. Small businesses simply never touch their robots.txt.
Result B: but 12.7% hand GPTBot a block signal while a normal browser gets the page fine. This is not noise — every re-checked case returned a stable 403 (a doctor, a bar, a café, a piano maker, a deli).
Why this inverts the scare story
The obvious assumption is "SMBs block ChatGPT on purpose." Wrong. They almost never do it deliberately — but their hosting blocks far more often, silently and unnoticed. Cloudflare Bot Fight Mode, a managed WAF rule, a site-builder default: the door shuts without anyone closing it.
The effect is harsh regardless: if OpenAI's crawler gets a 403, the page can't surface in ChatGPT. No SEO tool and no "AI-readiness score" shows you this — they audit the same empty skeleton the bot would get, and still hand you a green light.
Sample & method — open and reproducible
Source: OpenStreetMap via the Overpass API (free, no key, the query is reproducible from the tool). Frame: 9,232 unique domains — every OSM business with a website (retail, trades, office, food, health, hotel) across 8 mid-size German cities: Bonn, Freiburg, Erfurt, Regensburg, Münster, Heidelberg, Rostock, Kassel. (Kiel and Osnabrück timed out on Overpass and are honestly excluded.)
Measured sample: a deterministic systematic 1-in-23 draw → 402 domains, of which 346 answered a browser and 327 had a readable robots.txt. The list is fixed before any measurement — no cherry-pick.
Limits: only businesses with a listed website (skews slightly "more established"); 8 cities, not a national quota; a 403 to the GPTBot UA is often a generic bot rule, not a conscious anti-AI choice — the effect is identical. At n=346 the 95% confidence interval on the 12.7% figure is ±3.5 points. The number describes *this disclosed sample*, not "exactly X% of all German businesses".
FAQ
Does a 403 to GPTBot mean the business is deliberately blocking ChatGPT?
Usually not. Only 3.7% block AI crawlers on purpose via robots.txt. The 403 almost always comes from the hosting/CDN/WAF (e.g. Cloudflare bot protection) without the owner knowing. The effect is still that GPTBot cannot load the page.
Where does the sample come from?
From OpenStreetMap via the Overpass API: every business with a listed website across 8 mid-size German cities (9,232 domains), then a systematic sample of 402. Source open, tool in the repo, fully reproducible.
How do I check whether my own site is affected?
Quick self-test: fetch your homepage with the GPTBot user-agent, e.g. `curl -A "GPTBot/1.2" -I https://your-domain.com` — a 403/429 instead of 200 means your hosting blocks the crawler. For the visible effect, deeploupe's free checker shows whether ChatGPT & co. actually mention you. No signup.
As of July 2026