What is a user agent?
A user agent is the identifier a browser, crawler or other program sends with every request to a web server. In robots.txt the term also names the token a group of rules applies to, such as GPTBot.
The term has two meanings worth keeping apart. In HTTP traffic, the user agent is a string in the request header, often long and full of version numbers. In robots.txt, it is a short token at the start of a rule group. Crawlers match themselves to the group whose token fits their name.
The identifier is voluntary and easy to fake. Any program can claim to be GPTBot or Googlebot. That is why reputable operators also publish IP ranges that let you verify genuine fetches.
For AI visibility, the user agent is where blocks are decided. Firewalls and bot protection often filter on it. In our measurement across 346 reachable websites of German small businesses, 12.7 percent blocked the GPTBot user agent while the same request with a browser identifier went through.
What it means for your website
Request your homepage once with a browser identifier and once with an AI crawler’s user agent, then compare the status codes. If they differ, something is filtering by user agent. The deeploupe GPTBot check runs this comparison for GPTBot and ClaudeBot automatically and shows both responses side by side.
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card