SAFE and F1@K: Long-Form Factuality, Digested

SAFE comes from "Long-form factuality in large language models" by Wei et al., published at NeurIPS 2024 (arXiv 2403.18802). It tackles one question: how do you score the factual quality of a long answer that contains many separate claims?

The paper builds on an earlier line of work. FActScore (Min et al., EMNLP 2023) split a response into atomic facts and reported the share that a knowledge source supports. That is a precision measure. SAFE keeps the fact-level idea and adds a second axis, recall, so the score also reflects how much useful content an answer delivers.

Why this matters to a marketing lead: when an AI assistant describes your company, it produces exactly this kind of long, multi-claim answer. The scoring choices in SAFE are the same choices anyone faces when deciding whether ChatGPT or Perplexity "got your brand right".

SAFE sorts each fact in a response into one of three classes:

• Supported: the evidence backs the fact.
• Not supported: the evidence does not back it.
• Irrelevant: the fact does not bear on the question, so it is excluded from scoring.

From those counts the paper defines two numbers, where S is supported facts and N is not-supported facts:

• Precision = S / (S + N). Of the relevant facts stated, what share hold up.
• Recall against K = min(S / K, 1). Of the K supported facts a good answer should contain, how many did this one deliver. The cap at 1 means extra correct facts beyond K earn nothing more.

F1@K combines the two the way any F1 score does, as a harmonic mean. An answer scores well only if it is both accurate and sufficiently informative.

K is a choice the evaluator makes. It encodes how many supported facts you consider a complete answer for the task. Pick a different K and the recall term, and therefore F1@K, changes.

The finding that matters most for practitioners is a design finding, not a leaderboard. The authors state that FActScore's precision-only framing "does not penalize systems that abstain from answering too often or generate text with fewer facts."

Put plainly, a precision score rewards caution. A model that says two safe, true things scores the same precision as a model that says forty true things. A model that refuses most questions can look nearly flawless. Recall against K closes that gap by charging an answer for what it leaves out.

This page reports no headline accuracy figures or model rankings from the paper, because our evidence base does not record them. If you need those numbers, read them in the paper itself rather than in a secondhand summary.

The same concern shows up elsewhere in factuality work. Vectara's hallucination leaderboard, for example, excludes refused responses from its denominator and publishes an answer-rate column so that a high refusal rate cannot be mistaken for accuracy. Different method, same lesson: silence is not correctness.

SAFE is an evaluation method for general long-form answers. Keep its scope tight:

• It does not weight severity. Facts count equally. The FActScore paper says so directly: "every atomic fact has an equal weight of importance." A wrong acquisition claim and a wrong headcount cost the same.
• It is not about brands or AI search. It does not measure whether an engine retrieves or cites your site. The critical survey of 45 GEO studies found no technique with a stable, cross-platform causal effect on discoverability, and SAFE does not change that.
• It does not separate false from unverifiable. "Not supported" in this family of metrics covers claims the source contradicts and claims it simply lacks evidence for. FActScore defines its class exactly that way.
• K is not discovered. It is set by whoever runs the evaluation, so two F1@K scores are only comparable if they share K.

The honest takeaway: SAFE gives you a sound way to count. It does not tell you what to count as serious, or how to change what an AI says.

Say you audit what AI assistants say about your company across 50 buyer questions. Two failure modes from the SAFE paper apply directly.

A low error rate can hide one serious error

Average error rates dilute. One answer that names the wrong parent company can sit inside hundreds of clean answers and vanish in the mean. Report the worst error separately from the rate. 99Visibility rebuilt its own accuracy score on this lesson: it combines an error density with a size-independent worst-error term, so one critical error stays visible at 400 answers. Its constants were chosen by product judgement, not derived from data, and it says so.

A thin answer can look accurate

An assistant that says almost nothing about you has few errors to flag. That is the abstention problem SAFE's recall term exists to catch. Check whether answers contain the facts a buyer needs (price, what you sell, who it is for), not only whether stated facts are wrong.

Normalise by what you observed

Every established factuality metric divides by a count. Summing penalties without a denominator punishes bigger samples: under 99Visibility's retired formula, 50 prompts across 4 platforms with 20 minor flags (a 10% error rate) scored 0. Divide by answers that actually returned, not answers requested.

Next steps

1. List the 5 to 10 facts a correct answer about you must contain. That list is your K.
2. For each answer, count supported, wrong and missing facts.
3. Report rate, worst error and coverage together, per platform.

For how tools automate the extraction step, see how AI visibility tools detect factual errors. The free checker on this site runs a snapshot across up to three buyer questions if you want a starting sample.