Guides
Why AI answers vary: sampling, runs and confidence in GEO data
Repetition is the exception: in a 2,961-run study two AI answers gave the same brand list under 1% of the time, and ±5 points asks for about 300 answers.

By Mathias C.SEO consultant, and more recently GEO, at MentionBox
Ask an AI engine the same question twice and you get two answers, because every answer is a sample rather than a record pulled from a table. The study with the largest crowd behind it, run by SparkToro and Gumshoe with 600 volunteers putting 12 prompts through ChatGPT, Claude and Google's AI Overviews a combined 2,961 times in November and December 2025, puts the chance that two responses carry the same list of brands at under 1 in 100, and the chance of the same order at roughly 1 in 1,000. A single run is therefore a draw, not a measurement, and a GEO figure only becomes readable once enough draws sit behind it: about 300 answers per period for a margin of about ±5 percentage points.
This page is for whoever reads a visibility percentage in a monthly report, and for whoever writes the collection that produces it. It covers the five layers that make the answers move, what the published studies measured, which parts of an answer stay stable, the arithmetic that turns a pile of answers into a defensible figure, and how the measurement vendors handle repeated runs.
Five layers of variation, and who controls each
| Layer | What changes between two runs | Documented evidence | Who controls it |
|---|---|---|---|
| Token sampling | the wording, the length, the order of a list | OpenAI: "Chat Completions are non-deterministic by default (which means model outputs may differ from request to request)" | you, partly, on a model API |
| Server numerics | the output itself, with sampling switched off | Thinking Machines: 1,000 completions at temperature 0 produced 80 unique outputs, the most common appearing 78 times | the provider |
| Retrieval and fan-out | whether a search runs, which queries, which pages are read | Anthropic: 1 to 3 searches for a simple factual query, 10 or more for comparative research; cloro: mean domain overlap 0.40 between two runs minutes apart | the engine |
| The product in front of the model | the model version, the tools attached, the system instructions | Perplexity: API and web interface "share search infrastructure, but their tools, configuration, and models differ"; Google: AI Mode and AI Overviews "may use different models and techniques" | the vendor |
| The web and the locale | which pages exist and are indexed, which city the query came from | SE Ranking: 18% to 20% URL overlap across repeat runs of one general local query in AI Mode, 1.6% to 2% between cities | nobody |
Quotations from each source's own documentation, read on November 15, 2026: the OpenAI advanced usage guide, the Claude web search tool guide, the Perplexity FAQ and Google's AI features documentation. The study figures are dated in the sections below.
The layers compound, which is why a measured run-to-run gap is always wider than the sampling setting alone would suggest, and why no vendor or collection method removes the variation.
The sampler is the smaller half of the problem
Temperature is the part everyone knows: at a non-zero setting the model draws each token from a distribution, so the same prompt writes itself differently. OpenAI's guidance is that outputs "may differ from request to request", that a seed plus identical parameters makes them "mostly identical", and that "determinism is not guaranteed", with a system_fingerprint field published so you can see when the backend changed underneath you. The setting is also disappearing: Claude API version 2026-10-21, Anthropic's first new API version since 2023-06-01, states that "the temperature, top_p, and top_k request parameters are removed. A request that sends any of them returns a 400 error on every model." Nothing replaces them.
Switching sampling off would not deliver repeatability anyway. Thinking Machines published a test on September 10, 2025 that ran one prompt 1,000 times at temperature 0 against Qwen3-235B: 80 unique completions came back, the most common of them 78 times, with the first divergence at token 103. The cause sits in the server, which batches your request with whatever else arrives at that moment. Floating-point addition is not associative, so a different batch size changes the order of the additions and eventually flips a token. Rewriting the reduction kernels to be batch-invariant produced 1,000 identical completions in the same test, which is a research fix rather than a flag you can set on a commercial endpoint.
Retrieval moves the source list far more than the wording
For GEO the interesting variation sits one layer up, in the searches the engine runs before it writes. Anthropic's web search tool guide states that "simple factual queries typically use 1–3 searches" while "comparative or multientity research can use 10 or more", and that triggering is steerable from the system prompt. OpenAI's web search guide says the model "can choose to search the web or not based on the content of the input prompt". Two runs of one prompt can therefore differ by one answer written from the model's own knowledge and one answer assembled from eight pages.
cloro's published study of August 14, 2026 measured the consequence on 20 prompts sent to Claude Sonnet 5 with web search enabled: the mean overlap between the cited domains of two runs of the same prompt, minutes apart, was 0.40, with individual prompts spread from 0.06 to 1.00. Between two tiers of the same model family it fell to 0.22 to 0.25. Our explainer on query fan-out covers that step and which APIs return the queries themselves.
Geography adds its own spread. SE Ranking's volatility test of September 29, 2025 parsed 5,000 keywords 15 times each in Google AI Mode, three runs per city across five US cities. For general local queries the average domain overlap within one city was about 35% and the URL overlap 18% to 20%, while between two cities the URL overlap fell to 1.6% to 2%. Queries carrying an explicit location were steadier, at roughly 53% to 57% domain overlap within a city.
What stays stable, and what never does
The published studies agree on a split: the pool of brands an engine considers is reasonably steady, while the list it prints is close to unique each time.
| Measured object | Stability found | Study |
|---|---|---|
| Exact brand list, two responses to one prompt | under 1 in 100 for ChatGPT and Google's AI Overviews, slightly better on Claude | SparkToro and Gumshoe, 2,961 runs, November and December 2025 |
| Order of the list | about 1 in 1,000 runs before two lists match | same study |
| Length of the list | from 2 or 3 recommendations to 10 or more for the same prompt | same study |
| Leading brand, day to day | present on 82% of days on ChatGPT, 89% on Google AI Overviews | Detailed, 70,000+ responses, 1,300+ prompts daily over 28 days, September 23, 2026 |
| Core brands, those named on 80% of days or more | 13% of them churn from one day to the next on ChatGPT, 11% on AI Overviews | same study |
| Occasional brands | 78% churn day to day on ChatGPT, 65% on AI Overviews | same study |
| Cited domains, two runs minutes apart | mean overlap 0.40, range 0.06 to 1.00 | cloro, 20 prompts on Claude Sonnet 5, August 14, 2026 |
Figures as published by each study, read on November 15, 2026. The populations differ, so the rows describe one phenomenon at different grains rather than competing estimates of a single number.
Detailed's 28-day study adds the detail that makes rank reporting hard to defend: its most-cited domain appeared in 82% of ChatGPT responses, and came first in only 43% of them. In an analysis we ran in October 2026, 56.7% of the brands named for a question on one engine appeared in just one of its five runs, and 7.1% in all five.
So the figures that survive repetition are frequencies, computed over many runs: how often a brand is named, how often a domain is cited, how often an answer searches at all. The figures that do not survive are the ones taken from a single answer, starting with position. Our explainer on share of voice, visibility and position covers what each of those frequencies divides by.
The arithmetic that turns answers into a figure
A visibility percentage is a proportion measured on a sample, so it carries the usual band around it: about 1.96 × √(p(1−p)/n) at 95% confidence. For a brand named in 30% of answers, and for one named in 10%:
| Answers behind the figure | Band at 30% visibility | Band at 10% visibility |
|---|---|---|
| 10 | ±28.4 points | ±18.6 points |
| 30 | ±16.4 points | ±10.7 points |
| 72 | ±10.6 points | ±6.9 points |
| 100 | ±9.0 points | ±5.9 points |
| 300 | ±5.2 points | ±3.4 points |
| 1,000 | ±2.8 points | ±1.9 points |
Our arithmetic on the standard 95% interval for a proportion. The 300-answer and 72-answer rows match the roughly ±5 and ±10 point figures cloro publishes for the same sample sizes.
Two readings of that table decide most reporting arguments. A prompt run ten times carries a band of about ±28 points, so "present in 7 of 10 runs" is a fact about those ten runs and little else. And the band narrows with the square root of the count, so going from 300 to 1,000 answers buys 2.4 points of precision for three times the bill.
Showing a change costs more than showing a level. For a move from 20% to 30% at 95% confidence and 80% power, the usual two-proportion arithmetic asks for about 290 answers in each period, close to the 294 cloro publishes for that case. Smaller moves are far more expensive: halving the gap roughly quadruples the requirement, which puts a 5-point move at roughly 1,100 answers per side.
One caution the formula hides: answers are not independent draws. Five runs of one prompt resemble each other more than five runs of five different prompts do, because the prompt fixes the subject, the fan-out and much of the source pool. In our view the rule is to grow the prompt count and the run count together, and to discount a sample built from very few prompts below what the formula promises. Our guide on building a prompt set for GEO works that trade-off through with the cadence arithmetic.
How the measurement vendors handle repetition
Repetition is sold as cadence rather than as sample size, and the unit of billing is almost always one prompt, on one engine, for one run.
- Daily by default. Peec AI's setup documentation states that it "runs the prompt against AI models once a day and includes it in analytics". Promptwatch's data freshness page says that "the default, and what every plan uses, is daily", and advises that "for trend decisions, week-over-week comparisons of complete days beat reacting to a partial today".
- Repetition as a method. Evertune states on its own site that it "samples each prompt 100 times across every AI model", the highest published repetition figure in our records, checked on October 23, 2026.
- Runs as a request parameter. API for GEO exposes the count directly: its prompt tracking endpoint
POST /v1/jobs/executetakes amodelsarray and anexecutionsPerQuerynumber, billing is one credit per prompt, per model, per run at $13 per 1,000 answers falling to $7 at its largest tier, and reads over answers already collected are not billed, checked on October 23, 2026. Its price per answer runs higher than the scraping APIs', and its coverage of Gemini, Google AI Overviews, DeepSeek and Meta AI rests on our hands-on test of October 26, 2026 rather than on apiforgeo.com. MentionBox, which publishes this site, has a commercial link with API for GEO.
Free reads over stored answers decide more than they look on a noisy series, because the usual fix is to recount the answers you already hold under a stricter definition rather than to collect a larger month.
Five rules that keep a noisy series readable
- Fix the run count, then leave it alone. A period collected three times per prompt and a period collected once are not comparable, and the chart will show your scheduler rather than the engines.
- Print the denominator next to every percentage. The number of answers, prompts, engines and runs behind a figure is the only way a reader can tell a move from noise.
- Spread the runs across the period. A week of runs collected in one hour samples one state of the web and one state of the engine, which is also why Promptwatch advises comparing complete days.
- Keep the raw answers, their sources and their fan-out. A change of definition then costs a recount instead of a new collection, and "was our brand missing or was the answer brandless" stays answerable.
- Report a band, and grey out the cuts you cannot power. A 36-prompt monthly set holds at the topic level and collapses when filtered to one engine in one country, where the same percentage rests on a fraction of the answers.
Four questions settle most arguments about a GEO chart someone else built: how many answers is the percentage computed on, how many runs per prompt and did that number change, is the figure pooled across engines and weighted how, and is the move larger than the band. A series that cannot answer the fourth in one line is being read as a ranking rather than as an estimate.
Our view
Treat an AI visibility number the way a pollster treats a voting intention: a sample with a margin, reported at the level the sample supports and nowhere below it. The engines are built to produce a fresh answer every time, and the published studies put exact repetition under 1% for brand lists and under 3% for domain lists. Repeated runs give you a frequency instead, and frequencies move slowly enough to be managed: the pool of brands an engine considers for your category, the share of answers naming you, the domains it keeps returning to. Our ranking of 55 AI visibility APIs lists which of them return the raw answers a recount needs, and which hand back computed metrics only.
Frequently asked questions
Why does ChatGPT give a different answer to the same question?
Three layers add variation, and they stack. The model samples each token from a probability distribution, which OpenAI states plainly: "Chat Completions are non-deterministic by default". The engine then decides for itself whether to search the web and which queries to run, and OpenAI's web search guide says the model "can choose to search the web or not based on the content of the input prompt". Finally the assistant you use is a product with its own configuration, so two sessions can reach different models and different pages.
Does temperature 0 make AI answers reproducible?
No, and the measured gap is wide. Thinking Machines published a test on September 10, 2025 in which 1,000 completions of one prompt at temperature 0 produced 80 unique outputs, the most common appearing 78 times, with the first divergence at token 103; the cause is that server batch size varies with load and changes the order of floating-point additions. With batch-invariant kernels the same 1,000 completions came back identical. The lever is also disappearing from the APIs: Claude API version 2026-10-21 removed temperature, top_p and top_k, and a request that sends any of them returns a 400 error on every model.
How many times should I run the same prompt to measure AI visibility?
Several times per period, and the same number every period. SparkToro and Gumshoe, whose study had 600 volunteers run 12 prompts through ChatGPT, Claude and Google's AI Overviews a combined 2,961 times in November and December 2025, put the chance of two responses carrying the same brand list at under 1 in 100 and suggest asking "at least 60-100X". Vendors land in the same range by cadence rather than by repetition: Peec AI runs each prompt once a day, Promptwatch defaults every plan to daily, and Evertune states that it samples each prompt 100 times across every AI model.
How many answers do I need before a visibility change is real?
About 290 per period on each side of the comparison for a move from 20% to 30%, by the usual two-proportion arithmetic at 95% confidence and 80% power, which lines up with the 294 answers cloro publishes for the same case. For a single figure rather than a comparison, 100 answers put a 30% reading at roughly ±9 percentage points and 300 answers at roughly ±5. Anything built on ten answers per prompt carries a band of about ±28 points, which is wider than most of the moves a monthly report discusses.
Can I compare an AI answer from an API with one from the chat interface?
Not in the same series. Perplexity's own FAQ states that the API and the web interface "share search infrastructure, but their tools, configuration, and models differ, so results can differ", and that "the underlying AI model might differ between the API and the UI for a given query". Google documents the same split between its surfaces, saying AI Mode and AI Overviews "may use different models and techniques, so the set of responses and links they show will vary". Keep the collection method in the key of every stored answer, otherwise changing route reads as a change in visibility.
Is brand position inside an AI answer worth reporting?
It is the least stable figure in the set. SparkToro and Gumshoe measured roughly 1 in 1,000 runs before two lists arrive in the same order, and the same answers varied in length, from two or three recommendations to ten or more. Detailed's 28-day study of more than 70,000 responses found the most-cited domain appeared in 82% of ChatGPT responses yet came first only 43% of the time. Inclusion frequency over many runs holds up better than any rank.