We Scored AI Answers From a Single Sample for Four Versions
For the first four versions of our AI visibility scoring system, every number we produced came from running each buyer prompt exactly once per platform, and recording that single generated answer as the brand's score. AI answers are regenerated — and re-randomized — on every run. We were, in effect, grading brands on coin flips and reporting the results with two-decimal confidence. This post records what that error cost, how we finally caught it, and what changed in version 5. A companion research post explains the sampling standard itself; this one is the confession.
What versions 1–4 did
The pipeline was tidy. Take the client's buyer questions. Ask each one, once, on each platform. Record whether the brand was mentioned, whether its pages were cited, whether the answer recommended it. Divide by the number of questions, and you have rates — mention rate, citation rate, recommendation rate — clean numbers that fit in a table and survive being forwarded to a boss. The system computed quickly, cost little to run, and produced exactly the kind of output a dashboard wants.
Every step of that was defensible except the first one. An AI platform does not retrieve an answer; it generates one, sampling as it writes. Ask the same question twice and the sources rotate, the ordering changes, brands enter and leave. Kevin Indig's published research puts a number on it: run the same prompt three times and the cited sources are consistent only 2.2% of the time. We did not need his dataset to know something was off. We needed it to understand how far off we were.
What it cost
The damage of a noisy instrument is not that it is sometimes wrong. It is that you cannot tell which times. Fix priorities were set against single readings: a platform scored as a brand's weakest could simply be the platform where that day's one sample happened to exclude it, and effort flowed accordingly. A zero could mean invisible, or unlucky. A strong score could mean recommended, or fortunate. Client conversations were held over numbers whose error bars were larger than the effects we were discussing — and because the reports presented clean rates with no sample information, nobody downstream could see that. The numbers were not fabricated. They were worse than fabricated: they were real measurements of the wrong thing, the luck of one generation, presented as a property of the brand.
How we caught it
Not through an audit. Through annoyance. Re-running a fixed question during unrelated work returned an answer that flatly contradicted the recorded score — a brand marked absent, present and recommended; a platform marked strong, suddenly silent. Once is an anecdote. We re-ran the set systematically, and the scores moved like weather. That is the moment the method, not the data, became the suspect: if the same instrument gives different readings of an unchanged brand within the hour, the readings describe the instrument. Indig's 2.2% consistency figure, and his framing of prompt tracking as polling rather than rank tracking, gave us the vocabulary for what we had already tripped over. A poll with a sample size of one is not a poll.
What v5 changed
Version 5 of the scoring system rebuilt the sampling layer. In a full scan, every buyer prompt now runs three times per platform, and a brand's rate for a question is an appearance rate across the runs — 0 of 3, 1 of 3, 2 of 3, 3 of 3 — because never, sometimes, and always are different business situations that deserve different fixes. The question set stayed deliberately small and fixed: five to ten real buyer questions per measurement cell, because a fixed instrument is what makes month-over-month comparison honest. Single-run checks still exist for quick lookups, labeled as spot checks, never as measurements. And every report now carries a methodology note — sample size, question set, platforms, dates — printed next to the numbers it qualifies.
The uncomfortable part is the four versions before it. The fix took days. The error survived because the output looked like rigor: percentages, per-platform tables, tidy exports. Nothing about a coin flip announces itself when you only flip it once. If you take one practice from this post, take this one: whatever your GEO tool reports, ask how many times each prompt was run — and treat any answer of “once” as the answer to a different question than the one you paid to ask.
Sources & method. The version history (single-sample scoring in v1–v4; three-sample full scans, appearance rates, and methodology notes from v5) is from our own build and audit records for the tracking workspace used in this practice. The 2.2% three-run citation consistency figure and the polling framing are from published research by Kevin Indig (Growth Memo). The companion method post is “One Sample Lies: Why We Run Every Prompt 3 Times” in the Research section of this blog.
© 2026 Jun Liu · JUN & GEO. This article is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0) — https://creativecommons.org/licenses/by-sa/4.0/. You may share and adapt it with attribution and under the same license.