Do failed AI responses count against my visibility score?

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

They shouldn't. A failed response, such as a timeout, an API error or an empty reply, carries no evidence about your brand. A sound score leaves it out of the denominator and counts only answers that returned. Benchmark practice agrees: Vectara's hallucination leaderboard excludes refused responses and reports an answer rate alongside. The catch is sample size. Once failures thin out a platform's answers, its score can swing on a single flag.

Key takeaways

A failed request is missing evidence, so it should never count as either a miss or a clean answer.
Score answers that returned, then report the answer rate separately so you can see how much evidence each number rests on.
Vectara, FActScore, SAFE and RAGAS all divide by a count of real observations, and none sums raw penalties.
A per-platform score built on a handful of returned answers can swing hard on one flag, so treat it as noise until the sample grows.
Some answers return but were produced without web search, and those look like successes unless the tool records retrieval state.

The short answer: count what came back

A failed response is a request that produced nothing to score: an API error, a timeout, a rate limit, an empty reply. It tells you nothing about whether an AI platform names you or gets your facts right.

How a tool handles it changes your number in one of two wrong directions:

Counted as "not mentioned": your visibility falls for a reason unrelated to your brand.
Counted as a clean answer with no errors: your accuracy rises for the same non-reason.

The fix is a denominator rule. Score answers that returned, never answers requested. Then show the failures separately, as an answer rate, so the gap stays visible instead of hidden inside the score.

Say you track 20 buyer questions on four platforms, and one platform times out on 8 of them. That platform's mention rate should rest on 12 answers, and the report should say "12 of 20 returned". This matters because any blended number is an average of parts, as our guide to what AI visibility scores average explains. A wrong denominator on one part distorts the whole.

What benchmark practice does

Every established factuality metric divides by a count of observations. None of them sums penalties with no denominator.

Metric	What it divides by
Vectara HHEM leaderboard	Documents summarised, with refused responses excluded and an answer rate reported
FActScore (Min et al., EMNLP 2023)	Atomic facts in the response
SAFE / F1@K (Wei et al., NeurIPS 2024)	Supported plus not-supported facts
RAGAS faithfulness	Claims in the response

Vectara's design is the clearest model. It leaves refusals out of the accuracy rate, and its answer-rate column exists so a model that refuses often cannot pass off fewer attempts as accuracy. Exclude non-answers from the rate, then publish how many there were.

The SAFE paper adds a warning. Its authors introduced a recall term because precision alone "does not penalize systems that abstain from answering too often or generate text with fewer facts." Excluding non-answers from a rate is correct. Hiding them is not.

99Visibility rebuilt its accuracy formula on 18 September 2026 on this basis: the answer count is answers that returned, never answers requested. The earlier version had no denominator, which made scores fall as audits grew. That failure mode is covered in whether more prompts should change your score.

Why small samples swing

Excluding failures fixes the bias, but it shrinks the sample. When a platform returns only two or three answers, each answer carries a large share of the score.

In the same revised formula, one flag on two returned answers scores about 32. That reads as an accuracy collapse. It is really a partial run. For that reason the formula no longer reports a per-platform accuracy below five returned answers and falls back to the overall rate instead. Five is a product cutoff, not a statistical law, but the principle holds for any tool: a per-platform number needs a minimum sample before it means anything.

Small samples also amplify a second source of noise. Retrieval is per-query and discretionary on every platform, so the same question can be answered with live search one run and from the model's training data the next. Fewer answers means fewer chances for that mix to average out. If a platform's score moved sharply, check its answer count before you look for a cause. Our page on why an AI visibility score drops walks through the other usual suspects.

Edge cases: not every non-answer is a failure

Errors and timeouts

These produce no evidence. Exclude them from every rate and report them as a count.

Refusals and "I don't have information"

For accuracy, a refusal contains no facts to check, so it belongs outside the accuracy denominator, as Vectara treats it. For mentions, it is different. A delivered answer that doesn't name you is what a buyer would actually see. That is a real result and should count.

Answers that returned without web search

Some answers look like successes but were produced with no retrieval. Perplexity retired Sonar Chat Completions on 27 September 2026, and on its Agent API web search is opt-in. Code that was not migrated can return ungrounded answers with no error. Those answers reflect the model's training data, not your current site. Ask whether your tool records retrieval state per answer.

Wrong facts versus unverifiable ones

A returned answer that states a claim the tool cannot check is not the same as one that states a wrong price. Our guide to AI hallucinations about brands covers how to tell them apart.

Formula changes

Scores computed under different formulas are not comparable, even on the same brand. A careful tool records which version produced each score.

What to check in your own reports

Find the answer count per platform. Look for returned versus requested. If the report doesn't show it, ask.
Find out how failures are scored. Counted as misses, counted as clean, or excluded. Only the last is defensible.
Discount per-platform swings built on very few answers. Rerun before you act on them.
Read the answers, not just the number. A score can't tell you whether an answer was grounded or what it got wrong. See why a tool should show the full AI answers.
Compare tools only on the same denominator. Different failure handling is one reason AI visibility tools disagree about the same brand.

The free checker on this site runs up to three buyer questions, so treat its result as a snapshot. Read the answers themselves rather than treating one score as a trend.

Frequently asked questions

Is a refused answer the same as a failed response?

No. A failed response returned nothing, so it carries no evidence and belongs outside every rate. A refusal returned text. For accuracy scoring it has no facts to check, so benchmark practice such as Vectara's leaderboard leaves it out of the rate and reports it separately. For mention scoring, a delivered answer that doesn't name your brand is what a buyer would see, so it is a genuine result.

Why did one platform's score jump when nothing about my brand changed?

Check the answer count first. If failures cut that platform down to a few returned answers, one flag or one new mention moves the score sharply. Retrieval is also discretionary per query, so some answers use live search and others rely on training data. With a small sample that mix doesn't average out. Rerun the audit and compare answer counts before reading the change as real.

Should a tool retry failed requests?

Retrying is a reasonable way to rebuild the sample, as long as the tool records that it did so and scores only the answers that finally returned. What it should never do is fill gaps with assumed results. If retries still fail, the honest output is a smaller denominator plus a visible answer rate, not a padded score.

Can I compare scores from before and after a tool changes its formula?

Not directly. A score produced by an old formula and one produced by a new formula can differ for reasons unrelated to your brand, including how they handle failed or partial runs. Ask your tool which formula version produced each score. If it can't tell you, compare the underlying answers and their counts instead of the headline number.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.