A percentage is a compression. It takes a count and a denominator and produces a single number that reads as a rate. That is useful when the denominator is large and the estimate is stable; it is misleading when the denominator is three. A report that says "67% of answers mention you" from two of three runs borrows the authority of a survey while carrying three observations.
This article is about why the paid measurement runs every question three times, and why the result is written as "2 of 3" rather than as a percentage. It is a small design decision with a large honesty consequence.
Why three runs
Generative answers vary between runs even with the same prompt, the same model and the same day. The variation is the phenomenon, not noise to be averaged away: an engine that mentions a brand in two of three runs is in a different state from one that mentions it three times, and both are different from one that never does.
- One run cannot distinguish a stable mention from a coincidence, and a single failed call is indistinguishable from a real absence.
- Two runs can disagree, but a 1–1 split has no majority and no way to break the tie.
- Three runs are the smallest set with a majority and a visible split: 3 of 3 is stable, 2 of 3 is a majority with noise, 1 of 3 is a minority, 0 of 3 is absent.
The cost is the reason it stops there. Every run is a model call with a real price, and the value of the fourth and fifth run is smaller than the value of asking another question. Three is a floor chosen for interpretability, not a claim that three is enough for statistical precision.
Why the count is the result
| Reported as | What the reader sees | What it hides |
|---|---|---|
| 2 of 3 | The count, the denominator and the sample size in one glance. | Nothing relevant; the reader can compute the ratio if they want it. |
| 67% | A rate that looks comparable with surveys. | The denominator: three runs, one date, one model, one prompt. |
| "Often mentioned" | A qualitative impression. | Everything: no count, no denominator, no way to compare. |
| A score out of 100 | A single number that looks like a ranking. | The questions, the runs, the failures and the fact that it is not a ranking. |
The count is also the only form that survives aggregation honestly. If question A is 2 of 3 and question B is 1 of 3, the group is 3 of 6. Averaging the percentages (67% and 33%) gives 50%, which happens to equal 3/6 here but stops doing so as denominators vary. A report that stores counts can always reconstruct the totals; a report that stores percentages cannot.
What three runs can and cannot say
| Claim | Supported? | Why |
|---|---|---|
| "Mentioned in 2 of 3 runs on 12 March, on this model, for this prompt" | Yes | It describes exactly what was observed, with the conditions attached. |
| "67% of users see your brand" | No | Three runs are not a sample of users; they are three calls. |
| "Your visibility improved from 1 of 3 to 2 of 3" | Weakly | One run of difference on one question; it becomes a trend across a bank and a repeat. |
| "You rank third behind two competitors" | No | The method counts mentions, not positions; no ranking is measured. |
| "This question is unstable" | Yes, as a signal | A split across runs is evidence of variability; the count is what shows it. |
Failed runs are part of the count
A run can fail: a 429, a timeout, a refusal, a response that is not an answer. The honest report does not turn a failure into a zero and does not silently drop it. It records the failure, reports the completed denominator, and says which runs were excluded and why. A question with 2 completed runs is not the same as 2 of 3, and a question with 0 completed runs has no finding at all.
This is why the workbook matters as much as the prose. A row per run, with the raw answer and the failure code where there is one, is what lets a reader recompute the count. A summary that only shows 2 of 3 without the runs behind it is a number with no audit trail.
What this site does
The free scan reads a page; it does not measure mentions, and it does not claim to. The paid measurement is the part that runs a question bank: 25–30 questions, three runs each in a fresh session, answers recorded verbatim, counts written as x of n and never as a percentage. The deliverable spec says the same thing in one line: every number in the report has to be traceable to the rows in the workbook that produced it.
There is no score out of 100 for that measurement, on purpose. Three runs per question cannot support one, and a score would invite exactly the comparison the data cannot carry.
How to keep a count honest
# The shape of a run log: one row per run, never one row per question
question_id, run, mentioned, cited_url, error_code, answer_excerpt
Q1, 1, yes, example.com, , "..."
Q1, 2, yes, , , "..."
Q1, 3, no, , 429, ""
# The count is a group-by, not a stored percentage
SELECT question_id,
SUM(CASE WHEN mentioned = 'yes' THEN 1 ELSE 0 END) AS mentioned,
COUNT(CASE WHEN error_code = '' THEN 1 END) AS completed
FROM runs
GROUP BY question_id;Two rules make the table trustworthy. Keep the raw answer next to the label, so a disputed count can be checked against the text. And keep the failure code, so a denominator never silently shrinks.
What the count buys
It buys a claim that can be checked: the reader can see the denominator, the date, the model and the prompt, and can decide how much weight to put on it. It also buys the ability to combine questions without inventing a statistic. What it does not buy is precision, a ranking, or a prediction — three runs describe three runs.
The standard is the same as everywhere else in this series: publish the count, keep the denominator visible, and say what the measurement does not cover.