What data should a brand hallucination trend report include?
99Visibility Research · Updated October 21, 2026 · 5 min read
Short answer
A trustworthy brand hallucination trend report records, for every run, the answers that actually returned, the error rate per engine, the severity of each error, whether the engine searched the web, the prompt set version and the scoring formula version. Without those fields, a rising or falling line can reflect a bigger prompt set, a model swap or a new formula rather than a real change in what AI says about you.
Key takeaways
- An error rate needs a denominator, and that denominator should be answers that returned, not answers requested.
- Report error rates per engine, because each engine retrieves and generates differently.
- Record whether each answer used live web search, since grounded and ungrounded answers draw on different information.
- Version the prompt set, the brand fact set and the scoring formula, and never compare scores produced by different formula versions.
- Separate claims that contradict a known fact from claims you simply cannot verify.
The fields that make a hallucination trend trustworthy
A trend is only as good as the conditions held constant between runs. Each run in a report should carry these fields.
| Field | Why it matters |
|---|---|
| Run date and engine | Engines change models and behavior, so every number needs a date and a surface |
| Model identifier, where known | The model behind a consumer app can change without notice |
| Answers requested and answers returned | Failed or refused answers are not clean answers and must leave the denominator |
| Flagged answers and error rate per engine | A single blended rate hides which engine is wrong |
| Severity of each flag | A wrong acquisition claim and a wrong headcount are not the same problem |
| Error type | Contradicts a known fact, unverifiable, or outdated |
| Retrieval state | Whether the engine searched the web for that answer |
| Prompt set version | Adding or rewording prompts changes what you measure |
| Brand fact set version | More facts on record means more claims can be checked and flagged |
| Formula version | Scores from different formulas are not comparable |
The core metric is a hallucination rate: flagged answers divided by answers returned. Established factuality metrics all work this way. FActScore scores the share of atomic facts supported by a source, and RAGAS faithfulness divides supported claims by total claims. None of them sums penalty points without a denominator.
If a report you receive lacks most of these fields, treat its trend line as a picture of the tool's settings, not of your brand.
Why a formula change breaks the trend line
A scoring formula is part of the instrument. Change it and the old readings stop meaning the same thing.
99Visibility documented this in its own accuracy research on 2026-09-18. Its first formula subtracted penalty points from 100 with no denominator. At a 10 to 12.5% error rate, a 16-answer audit scored 90, a 68-answer audit scored 65, and a 200-answer audit scored 0. A customer who added prompts watched the score fall for buying more coverage. The revised formula divides by answers returned, so the same error rate scores the same at any sample size. Scores written before that date are recorded as a legacy version and are not comparable with newer ones.
What this means for you:
- Ask every vendor which formula version produced each score.
- Break the chart at any version change, or recompute old runs under the new formula.
- Be wary of any trend that improves right after a methodology update.
Scoring constants are often chosen by product judgement, not derived from data. A good report says so. The same applies to what belongs in any AI SEO report: label which numbers are measurements and which are the vendor's model.
Edge cases: small samples, refusals and unverifiable claims
Small samples
One flag on two returned answers looks like a collapse. It is really a partial run. 99Visibility's accuracy research stops reporting per-platform accuracy below five returned answers for this reason. Look for a minimum-sample rule in any report.
Refusals and errors
The Vectara hallucination leaderboard publishes an answer-rate column so a high refusal rate cannot pass as accuracy. Your report should show answers requested next to answers returned.
Severity versus rate
A pure average lets one serious error fade as the sample grows. The SAFE paper added a recall term because precision alone does not penalize systems that say too little. A good report shows the worst error per run alongside the rate. Here's why one serious error can outweigh many small ones. Note that severity tiers are a product decision: FActScore gives every atomic fact equal weight.
Unverifiable claims
If an AI states your headcount and no headcount is on record, the claim is unverified, not proven false. FActScore's not-supported class mixes both cases. Keep them in separate columns so you fix real errors first.
How to audit a report you already have
- Check the denominator. Confirm the rate divides by answers returned.
- Check per-engine rows. A blended number hides the engine causing trouble.
- Find the version fields. Prompt set, fact set and formula changes should be marked on the chart.
- Check retrieval state. Without it, you can't tell stale training data from a bad retrieved source.
- Read the flagged answers themselves. Then trace which source caused the wrong fact.
The free checker on this site gives a one-time snapshot for three buyer questions. It is a starting point, not a trend. A trend needs repeated runs under fixed, recorded conditions.
Frequently asked questions
How often should hallucination data be collected to show a trend?
There is no published standard interval. What matters more is consistency: the same prompt set, the same engines and the same formula on each run. Record the date of every run so you can line up changes in the error rate with model launches, site updates or changes on third-party pages. A long run of comparable snapshots is more useful than frequent runs under shifting conditions.
Can I compare hallucination rates across two different tools?
Only with care. Tools differ in prompts, fact sets, retrieval settings, severity weights and formulas. Two tools can report different rates for the same brand in the same week and both be internally consistent. Compare each tool against its own history, and only compare across tools when both disclose their denominators, retrieval state and formula versions.
Does a falling hallucination rate mean my fixes worked?
Not by itself. A falling rate can come from a new formula, a larger prompt set, more refusals, a model update or more grounded answers. Check the version fields and retrieval state first. If those held steady and specific wrong claims stopped appearing, that is a stronger sign. Nobody controls what the models say, so treat it as an observation, not proof.
Why do citations in a flagged answer sometimes look correct?
A citation can resolve and look relevant without supporting the claim beside it. OpenAI's documentation notes that engines often retrieve more sources than they cite, and research on attribution shows citation output is partly separate from source use. Store the cited URLs with each flagged answer and read them, rather than assuming the citation explains the error.
Sources
- [2305.14251] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- [2403.18802] Long-form factuality in large language models
- GitHub - vectara/hallucination-leaderboard: Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documen
- Faithfulness - Ragas
- Searching the web with ChatGPT
- AI Features and Your Website
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.