What is answer variance in AI search?
Answer variance is the fact that a language model gives different answers to the same question when asked again, with different wording, different sources and different named brands. It is the reason a single query cannot support a stable statement about visibility.
Part of the variation is intended: when generating text, models choose among several likely continuations. But even with settings meant to switch that off, answers do not stay identical. Thinking Machines Lab asked the same prompt 1,000 times at temperature 0 and received 80 different completions; the study attributes this to numerical effects of batched processing on GPUs.
AI search engines add a second source of variance. The web search running in the background returns different results depending on time, phrasing and index, which changes the cited pages and the named vendors as well.
For measurement this means: a mention in one run is hard evidence that the engine can name the brand. A missing mention in one run, however, does not prove that it never does.
What it means for your website
Design measurements so that each question is asked several times, and report frequencies instead of single results. Never compare one run from today with one run from last month and read the difference as a trend. If you only have one run, label it as a snapshot, as deeploupe’s free ChatGPT check does.
More on deeploupe
Sources
Terms help you understand. Whether AI crawlers can reach your site is something you measure.
Check your website for freefree · no signup · no credit card