Why can't I compare old and new AI visibility scores?

A score is not a reading of your brand. It is a calculation over a sample of AI answers. Three things feed it: the formula that turns answers into points, the models that wrote the answers, and the prompts that were asked. Change any one and the number moves.

That means a drop from 72 to 61 has at least three possible causes:

• The formula changed. The tool reweighted a component or fixed a bug.
• The models changed. A vendor shipped a new default, or the tool started calling a different model.
• Your brand's answers changed. This is the only one you care about.

If the tool does not tell you which formula and which models produced each score, you cannot separate the third cause from the first two. So the rule is simple: compare scores only within one formula version. Treat every version change as a break in the line, not a point on it. Our guide to what an AI visibility score averages covers the inputs in more detail.

On 18 September 2026, 99Visibility replaced the accuracy component of its score, which carries 30% of the total. The old version subtracted penalty points from 100 with no denominator. Its internal write-up shows what that did at a constant 10% minor-error rate:

The brand's accuracy did not change. The sample size did. A customer who added prompts watched the score fall for buying more measurement. This is why adding more prompts should not change your score on its own.

The published factuality metrics avoid this. FActScore divides supported facts by total atomic facts in a response. SAFE pairs a precision rate with recall against a target count, because a rate alone lets one serious error vanish in a large sample. Vectara's leaderboard excludes refused responses from its denominator. The replacement formula normalises by answers that actually returned and keeps the single worst error visible. Its weighting constants were chosen by product judgement, not derived from data, and the write-up says so.

The consequence is plain: every accuracy score written before that date is a v1 score and cannot be compared with a v2 score. Each audit now records which formula version produced it.

A formula can stay fixed while the answers underneath it shift. AI vendors change defaults often. As of a September 2026 snapshot, the Gemini API had moved two flash generations past what one measured column called, and ChatGPT's plans served different models by tier. Google says AI Overviews and AI Mode "may use different models and techniques", so even one vendor is not one surface.

Knowledge cutoffs move with the model. OpenAI lists a 16 February 2026 cutoff for the GPT-5.6 family and 31 August 2025 for the Instant path. A model swap can change what a retrieval-off answer knows about you overnight. See whether a new model release changes what AI says about your brand and how knowledge cutoffs keep old news alive.

So a score history needs two version markers, not one: the formula and the measured models.

Some score movements are artefacts even within one version.

• Partial runs. If a platform errors and returns two answers instead of twenty, one flag can read as a collapse. A careful tool suppresses per-platform figures below a minimum answer count. 99Visibility now withholds per-platform accuracy below five returned answers.
• Backfilled history. When a tool relabels old rows, failed audits should stay blank. A score that was never computed should not acquire a version.
• Retrieval state. Engines decide per query whether to search the web. Any score averages answers with and without live retrieval, and that mix is not observed.
• Different tools. Two vendors use different formulas, prompts and models. Their numbers are not two readings of one quantity. That is why AI visibility tools disagree.

Before you put a score history in a board deck, ask the vendor for these:

1. A formula version on every stored score, visible in exports.
2. A dated changelog that says what changed and why.
3. The models called on each audit, with search on or off.
4. The prompt set version, so added or edited prompts are visible.
5. The returned answer count per platform, not just prompts requested.
6. Raw answers retained, so old audits can be rescored under a new formula if needed.

Then act on it. Draw a vertical line on your chart at every version change. Start a fresh baseline after each one, using the steps in setting an AI visibility baseline. When you report to leadership, show the break rather than smoothing over it. Explaining score limits to stakeholders covers how to frame that.

Can old scores be converted to the new formula? Only if the tool kept the raw answers and flags from each old audit and reruns them through the new formula. There is no reliable multiplier between versions, because the change is not linear: under a formula without a denominator, the gap depends on how many answers each audit checked. If raw data is gone, treat old scores as a separate series and start a new baseline.

Why did my score fall after I added more prompts? If your error rate stayed the same, the formula is probably the cause. A formula that sums penalties without dividing by answer count punishes larger samples. A sound formula gives the same score for the same error rate at 20 answers or 400. Ask your tool whether its accuracy component is normalised by returned answers.

Does a new AI model release reset my baseline? It should, if the tool switches to the new model. A new default can change knowledge cutoffs, search behaviour and wording at once. Your brand did nothing, but the answers shifted. Mark the date the tool's measured models changed and compare scores only on each side of it.

Is a higher score after a formula change good news? Not by itself. A formula change can raise every customer's score at once. Look at the underlying findings instead: which specific facts each engine got wrong, and whether those errors are gone in the same-version audits that followed. Fixed facts are evidence. A higher number across a version break is not.