Should adding more prompts change my AI visibility score?

Adding prompts should make your score more stable, not lower. More answers give a better estimate of the same underlying rate.

A score can legitimately move when the new prompts ask different things. Say you add unbranded comparison questions to a prompt set that was all branded. The engines may name you less often for those, so the score reflects real coverage. That's a change in what you measure.

What should never happen is a drop caused by arithmetic. If 10% of answers contain an error at 20 prompts and 10% contain an error at 200, the accuracy part of the score should land in the same place. When it doesn't, the formula is counting errors in absolute terms. Every extra prompt is another chance to lose points, and nothing offsets it.

The next move: before you compare a score across plan sizes, find out whether it is a rate or a running total.

99Visibility found this defect in its own accuracy component and published the arithmetic. Its first formula subtracted fixed penalty points from 100 for each flagged answer, with no denominator. Run on real audit shapes, it produced this:

The 68-answer audit was cleaner than the 16-answer one and scored 25 points lower. At 200 answers, a 10% minor error rate zeroed the component. A customer who bought more prompts watched their score fall for buying a bigger plan.

The replacement divides capped per-answer penalties by the number of answers that returned. It also keeps a separate term for the single worst error. That way one critical mistake stays visible at 400 answers instead of averaging away. In the published comparison, 400 answers with one critical hallucination scored 65 under the old formula and under 80 under the new one.

Two honest limits come with that fix. The curve constants were chosen by product judgement, not derived from a dataset. And scores from the two formulas are not comparable, which is why old and new AI visibility scores can't be compared directly.

Every established factuality metric divides by a count. None sums raw penalties.

• FActScore (Min et al., EMNLP 2023) scores the share of atomic facts in a response that a source supports.
• SAFE (Wei et al., NeurIPS 2024) adds recall against a target count, because precision alone doesn't penalize a system that says very little.
• RAGAS faithfulness divides supported claims by total claims.
• Vectara's hallucination leaderboard reports a rate per document and shows answer rate separately, so refusals can't pass as accuracy.
• HaluEval 2.0 reports hallucinated statements over statements, and responses with at least one hallucination over responses.

One thing the research does not back: severity weighting. FActScore gives every atomic fact equal weight. A vendor that grades errors as critical, moderate or minor is making a product choice. That can be a reasonable choice, but it isn't benchmark practice, and the vendor should say so.

Some changes alongside a bigger sample are real and worth understanding.

• Small samples are noisy. One flag on two returned answers reads as a collapse. A sensible tool suppresses a per-platform rate below a minimum answer count. See how many prompts you need for sizing.
• Failed answers. If an engine times out, that answer returned no evidence. It should leave the denominator, not count as clean. More on failed AI responses.
• Retrieval mixes. Engines search the web at their own discretion per query. Any blended score averages answers given with and without live retrieval, so new prompts can shift that mix. Our guide on what AI visibility scores average covers this.
• Samples undercount. In Otterly's experiment, Microsoft's own report counted 42,817 citations for one site against 3,934 logged by a prompt tracker. A prompt set is a sample, whatever its size.

No vendor controls what the engines say. A good score tells you what it measured and how.

Put these to any tool before you trust its trend line:

1. Is each component a rate or a running total? What is the denominator?
2. Does the denominator count answers requested or answers that returned?
3. If I double my prompts at the same error rate, does the score stay put?
4. How does one severe error show up in a large sample?
5. Is severity weighting used, and is it described as your choice?
6. What is the minimum answer count before a per-platform number is shown?
7. Which formula version produced each stored score, and are old scores relabelled when it changes?
8. Does the score separate answers given with and without web retrieval?

Then lock a baseline before you change anything on your site. The AI visibility baseline playbook walks through it.

Why did my score drop after I upgraded plans? Check two things. First, whether the new prompts cover topics where engines name you less, which is a real coverage change. Second, whether the formula sums penalties instead of dividing by answers returned. If your error rate stayed flat and the score still fell, the second cause is likely. Ask the vendor to rerun your old prompt set alone and compare.

What does a normalized accuracy score mean? It expresses errors as a share of observations, such as flagged answers over answers returned. That makes the number independent of sample size. FActScore, RAGAS faithfulness and Vectara's leaderboard all work this way. A normalized score can still add a separate term for the worst single error, so one serious mistake isn't diluted by a large sample.

Is a bigger prompt set always better? A bigger set narrows noise, but only if the prompts reflect what buyers actually ask. Engines rewrite queries before searching, so no prompt set sees the real retrieval input. Size helps precision. Relevance decides whether you're measuring the right thing. Pair a reasonably sized set with branded and unbranded questions you'd actually want to win.

Can I compare scores from two different vendors? Rarely. Vendors pick different engines, prompts, denominators, severity weights and retrieval settings. Two numbers on the same 0 to 100 scale can measure different things. Compare each vendor against itself over time, using a fixed prompt set and a known formula version, and treat cross-vendor gaps as a prompt to ask how each one is built.