FActScore: How Factual Precision in AI Answers Is Measured

FActScore comes from Min et al., published at EMNLP 2023 under the title "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation" (arXiv 2305.14251).

The problem it targets is simple to state. A long answer from a language model mixes true and false statements. Grading the whole answer as right or wrong throws that detail away. FActScore grades each claim instead.

For a brand team, this is the same problem in a different setting. An AI answer about your company might get your founding year right, your pricing wrong and your headcount unverifiable, all in one paragraph. A definition of FActScore is useful because it gives you a vocabulary for that mix.

FActScore works in three steps.

1. Split the response into atomic facts: short statements that each carry one piece of information.
2. Check each atomic fact against a knowledge source and mark it supported or not supported.
3. Divide the supported facts by the total facts in the response.

Written as a formula, the score for a response y is f(y) = (1/|A_y|) Σ 𝕀[a supported by C]. A_y is the set of atomic facts in the response, C is the knowledge source, and the indicator counts 1 for each supported fact.

Two design choices matter most.

• The denominator is the count of facts. The score is a proportion, so a 10-fact answer and a 100-fact answer sit on the same scale.
• Every fact weighs the same. The paper says outright that "every atomic fact has an equal weight of importance". A wrong acquisition claim and a wrong office address each cost one fact.

This puts FActScore in good company. RAGAS faithfulness divides supported claims by total claims. Vectara's hallucination leaderboard reports a rate over documents summarised. HaluEval 2.0 defines rates over statements or responses. None of these sums raw penalties.

The evidence base behind this library does not record FActScore's benchmark results for specific models, so this page does not quote any. What it does record are the properties of the metric itself, and those are the useful part for a brand team.

Equal weighting is a stated choice

The authors chose equal weights on purpose and said so. Anyone who adds critical, moderate and minor tiers is making a product decision. That can be the right call, but it is not inherited from the benchmark.

Not-supported covers two cases

FActScore's not-supported class is defined as "the source contradicts or lacks evidence for" a claim. It does not separate the two. A claim your records prove false and a claim you simply have no record of both land in the same bucket.

Precision alone has a blind spot

The follow-up SAFE and F1@K study (Wei et al., NeurIPS 2024) noted that a precision-only framing "does not penalize systems that abstain from answering too often or generate text with fewer facts." A cautious answer that says almost nothing can score perfectly. SAFE added recall against a target number of facts to close that gap.

FActScore measures one thing well and leaves several things open.

• It does not tell you why an answer is wrong. It flags unsupported facts. It says nothing about whether the error came from stale training data, a third-party page the engine retrieved, or a contradiction on your own site.
• It is only as good as the knowledge source. A fact is "supported" relative to C. If your reference set is incomplete, true claims get marked not supported.
• It does not rank errors by harm. By design, one fact is one fact. A brand that cares more about a wrong price than a wrong founding month has to add that judgement itself.
• It does not separate false from unverifiable. See above. For a brand team, that distinction decides the response: a contradiction needs a correction, a gap may need a published fact.
• It says nothing about AI visibility. FActScore measures the precision of text a model produced. It does not measure whether an engine mentions you, cites you, or retrieves your pages.

Treat it as a scoring pattern, not as evidence about how any AI platform behaves.

The core lesson is about the denominator. Count errors against the number of facts or answers you checked. Do not subtract penalty points from 100.

Say you score AI answers about your company by taking 15 points off for each flagged claim. Check 16 answers and find 2 flags: you score 70. Expand to 68 answers at a lower error rate and find 7 flags: 105 points of penalty zeroes the score. The answers got cleaner and the number collapsed. A rate over answers checked would have shown the improvement.

Here's how to apply the FActScore pattern to your own checks.

1. List the facts you can verify. Pricing tiers, founding year, headquarters, product names, ownership. Fix any contradictions on your own site first, or your reference source is unreliable.
2. Break each AI answer into claims and check each claim against that list.
3. Report a rate: wrong claims over claims checked, or answers with at least one error over answers returned. Count only answers that actually came back.
4. Track the worst error separately. A mean dilutes one serious error as the sample grows. SAFE exists partly because of that.
5. Split contradicted from unverifiable. FActScore doesn't, but your response differs. Contradictions need correcting. Gaps are a prompt to publish specific, fetchable facts, which gives an engine a sentence it can quote, though no one can promise it will.

For a fuller look at how checking tools apply this, see how AI visibility tools detect factual errors.

What is an atomic fact in FActScore? An atomic fact is a short statement that carries a single piece of information, such as "the company was founded in 2015". FActScore breaks a long response into these units so each can be checked on its own against a knowledge source. The score is the share of atomic facts that the source supports, which gives a finer reading than grading a whole answer as right or wrong.

Does FActScore weight serious errors more heavily? No. The paper states that every atomic fact has an equal weight of importance. A wrong price and a wrong office address each count as one unsupported fact. If your team needs severity tiers, add them as an explicit product decision and document why. Other common factuality benchmarks, including Vectara's leaderboard, also score without severity weights.

How is FActScore different from SAFE? FActScore measures precision: the share of claims that are supported. SAFE, from Wei et al. (NeurIPS 2024), added recall against a target number of facts, because precision alone does not penalize answers that abstain or say very little. Used together, they show both how accurate an answer is and how much it actually told you.

Can I use FActScore to measure what AI says about my brand? You can borrow its structure, not run the benchmark as is. Build a reference list of verified brand facts, split each AI answer into claims, check each claim against the list, and report the supported share. Add a separate flag for your worst error and split contradicted claims from unverifiable ones, which FActScore does not do. For causes of wrong answers, see our guide to [AI hallucinations about brands](/guides/ai-hallucinations-about-brands).

Why does the contradicted versus unverifiable split matter? Because the fix differs. A contradicted claim, like a wrong price, means the AI stated something your records prove false, and the next move is to find and correct the source. An unverifiable claim means you have no record to check it against. The next move there is to confirm the fact and state it on a page AI crawlers can fetch.