FActScore: definition and how it works

FActScore (Factual precision in Atomicity Score) measures how much of a long-form answer is backed by a knowledge source. The paper by Min et al. (EMNLP 2023) defines it for one response as the number of supported atomic facts divided by the total number of atomic facts in that response.

Three choices define the metric:

• The unit is the atomic fact. A short statement that carries one piece of information.
• The judge is a fixed source. A fact counts as supported only if that source backs it.
• The denominator is the response's own facts. The score is a proportion, not a sum of penalties.

This entry stays short. For the full method and its limits, read how FActScore measures factual precision.

The calculation runs in four steps:

1. Generate a response, for example a biography or a product description.
2. Break it into atomic facts. "The company was founded in 2019 in Austin" becomes two facts.
3. Label each fact supported or not supported against the knowledge source.
4. Divide supported facts by total facts.

Two design decisions shape every result.

Equal weighting. The paper states that "every atomic fact has an equal weight of importance." A minor detail and a claim that would change a buyer's decision move the score by the same amount.

A mixed failure class. A fact is not supported when the source contradicts it or lacks evidence for it. So a provably false claim and an unverifiable one land in the same bucket. Faithfulness in RAGAS works the same way: supported claims over total claims, with unsupported claims counting against the answer.

AI platforms get specific facts wrong about specific brands. Any tool that reports an accuracy figure for those answers faces the same questions FActScore answers.

• Normalize by a count. Every established factuality metric divides by something: facts, claims, statements or responses. A score that only subtracts penalty points will fall as you ask more questions, even if the error rate stays flat.
• Decide what not supported means. A wrong price is a contradiction. An employee count you never published is unverifiable. Treating them the same is a choice, and you should know which choice your tool made.
• Treat severity as a product decision. Weighting a wrong acquisition above a wrong headcount is reasonable. It is not inherited from FActScore, which weights equally.

The next move: when a vendor shows you a factual accuracy number, ask for its denominator, its source of truth, and how it handles unverifiable claims. Our explainer on how AI tools detect factual errors covers what to ask.

• "A high FActScore means the answer is complete." It doesn't. Precision ignores what the answer left out. The SAFE paper notes that precision-only framing "does not penalize systems that abstain from answering too often or generate text with fewer facts." Its F1@K metric adds recall against a target number of facts. See SAFE and F1@K.
• "Not supported means false." It can also mean the source is silent.
• "It measures truth." It measures agreement with one knowledge source. If that source is outdated, the score inherits the error.
• "It is an AI visibility score." It says nothing about whether an engine names or cites you. Accuracy is one input that blended visibility scores may average with others.

• Atomic fact: the unit FActScore counts.
• Faithfulness: the RAGAS metric for claims supported by retrieved context.
• Factual accuracy: the broader idea of whether AI answers about a brand are correct.
• SAFE and F1@K: a later method that pairs precision with recall.
• Hallucination rate: HaluEval 2.0 defines a statement-level rate (hallucinated statements over statements) and a response-level rate (responses with at least one hallucination over responses).

What is the difference between FActScore and SAFE? FActScore is precision only: supported facts over all facts in a response. SAFE, from Wei et al. (NeurIPS 2024), keeps a precision term but adds recall against a target number of facts, K, and combines them as F1@K. The change exists because a short, cautious answer can score perfect precision while telling the reader almost nothing. SAFE penalizes that.

Does FActScore weight serious errors more heavily? No. The paper states that every atomic fact has equal weight. A wrong founding year and a wrong claim that a company was acquired each remove one supported fact. If you need to separate serious errors from trivial ones, you have to add severity weights yourself, and that is a product decision rather than standard benchmark practice.

Can I use FActScore to check what ChatGPT says about my company? You can borrow the method. Split the answer into atomic facts, check each against a source of truth you control, such as your verified pricing and company facts, and compute the supported share. Your result is only as good as that source. Keep it current, and record which claims were contradicted versus simply unverifiable.

Why does the denominator matter so much? Without one, a score depends on how many questions you ask. Summing penalty points means a larger sample produces a lower score at the same error rate. Dividing by facts, claims or responses makes the result comparable across sample sizes. FActScore, SAFE, RAGAS faithfulness and Vectara's hallucination rate all normalize by a count.