Hallucination rate: definition and how it's measured

Researchers use two versions of the rate, and the HaluEval 2.0 study names both.

• Statement level (micro rate): hallucinated statements divided by all statements checked. One long answer with ten claims contributes ten observations.
• Response level (macro rate): responses containing at least one hallucination divided by all responses checked. One bad claim marks the whole answer.

The response-level rate is usually higher, because a single error in a long answer counts as a failed answer. Always state which one you are reporting.

Every established factuality metric divides by a count. None just adds up penalties.

• FActScore splits a response into atomic facts and reports the share supported by a knowledge source. It states that every atomic fact has equal weight. Its not-supported class covers claims the source contradicts or simply lacks evidence for. Our FActScore explainer covers it in depth.
• SAFE adds a recall term because, its authors note, precision alone doesn't penalize systems that abstain or say less.
• RAGAS faithfulness divides supported claims by total claims in a response.
• The Vectara hallucination leaderboard scores summaries of short documents, defines hallucination rate as 100 minus a consistency rate, and publishes a separate answer-rate column.

Why refusals leave the denominator

A refused or failed answer produced no claims to check. Counting it as a clean answer rewards silence. Vectara's answer-rate column exists for exactly this reason. The right move is to exclude non-answers from the rate and report the answer rate beside it.

For a brand, being named is only half the result. An answer that names you with the wrong price or a discontinued product can do more harm than no mention. The kinds of brand facts AI gets wrong tend to be specific: prices, ownership, product names.

Two design choices decide whether a brand-level rate is useful:

• A real denominator. A score that subtracts penalty points without dividing by answers falls as you ask more questions, even when the error share stays flat.
• Separate false from unverifiable. An answer that contradicts your published price is wrong. An answer stating a headcount you never published can't be checked. FActScore lumps these together, so splitting them is a design decision, not a research rule.

If you track this over time, see what a hallucination trend report should include. Your control is on your side: state facts consistently on pages crawlers can fetch. What the model says remains outside your control.

• "A leaderboard rate tells me how accurate a model is about my brand." Vectara measures summarization of supplied documents. That task differs from answering open questions about a company.
• "Lower is always better." A model that refuses often can post a low rate. Read the answer rate too.
• "Rates from two benchmarks can be ranked." Tasks, judges, units and the treatment of unsupported claims all differ. A 5% in one benchmark and a 5% in another are not the same quantity.
• "Severity weighting is standard." FActScore weights all facts equally and Vectara is binary per summary. Any critical-versus-minor weighting is a product choice and should be labelled as one.

• Hallucination vs outdated answer: a stale fact from training data is a different failure with a different fix.
• Citation rate: how often an engine cites a source, a separate question from whether its claims are right.
• How often AI search gets facts wrong: what the published measurements show.
• Which AI platform is most accurate about brands: why accuracy should be measured per platform.

What is a hallucination leaderboard? A hallucination leaderboard ranks models by an error rate on a fixed benchmark. Vectara's well-known leaderboard asks models to summarize short documents and checks whether the summaries stay consistent with the source. It reports a hallucination rate alongside an answer rate. The ranking is only valid for that task, so treat it as one signal about summarization, not a verdict on how a model describes your brand.

Is hallucination rate the same as error rate? Mostly, but definitions vary. Error rate is the general term for wrong outputs over total outputs. Hallucination rate usually means claims that are false or unsupported by a reference source. Some metrics, like FActScore, count a claim the source has no evidence for as not supported, which is broader than provably false. Check how each source defines an error before comparing numbers.

Should I use the statement-level or response-level rate? Use both if you can. The statement-level rate shows how much of the content is wrong. The response-level rate shows how often a reader meets at least one error, which better reflects what a buyer experiences. For brand monitoring, the response-level view often maps more closely to risk, because one wrong price in an answer is enough to mislead.

How many answers do I need for a reliable rate? There is no published threshold that fits every case. The practical rule is that tiny samples swing wildly: one flagged error in two returned answers looks like a collapse. Report the number of returned answers next to any rate, and avoid per-platform rates when only a handful of answers came back. Count answers that returned, not answers you requested.