Building an AI Error Log and Correction Tracker

AI engines state specific facts about specific brands, and some of those facts are wrong. A wrong price, an outdated product name or a competitor's feature credited to you can reach a buyer before your site does. A spreadsheet of screenshots does not fix any of that.

The tracker turns each wrong answer into a work item. It answers three questions: what is wrong, where did the engine probably get it, and has the fix held. It also gives you one honest number to report upward.

Keep the scope tight. Log factual claims about your brand (prices, plans, features, founding year, ownership, locations, integrations). Leave tone and sentiment for a separate view. Use the guide on what counts as a critical AI error to decide which rows need action this week.

One row per wrong claim, not per answer. A single answer with two wrong facts gets two rows that share an answer ID.

Three columns need care.

Engine and surface

Do not merge Google surfaces. Google says AI Overviews and AI Mode may use different models and techniques, and they return different links. The Gemini app is a third product.

Retrieval state

As of October 2026, ChatGPT decides per query whether to search, and Claude.ai search is a user toggle. If you saw citations, the answer used search. If you saw none, write "unknown", not "ungrounded". An unknown answer may come from a model's trained knowledge, which has a fixed cutoff and differs by model.

Error type

Split "contradicts a fact on record" from "unverifiable". An engine that states a headcount you never published is not the same problem as an engine that quotes the wrong price.

Run this workflow each cycle.

1. Fix your prompt set. Write the buyer questions once and reuse them word for word. Changing prompts between runs makes trends meaningless.
2. Control the session. Run logged out, or with memory and custom instructions off. OpenAI says ChatGPT may use saved memories when it rewrites a search query, so a logged-in run is not reproducible.
3. Capture the full answer. Save the text, every cited URL and a screenshot. Note whether the platform returned an answer at all.
4. Check each factual claim against your fact sheet. Keep one internal list of true values (prices, plan limits, founding year, HQ) with the page that states each one.
5. Log a row per wrong claim. Quote it exactly. Assign type and severity.
6. Trace the suspected source. Open the cited URLs and search them for the wrong value. Then check your own site: the same fact stated differently on the pricing page, the about page and the footer is a common cause. The walkthrough on finding which source caused a wrong AI fact covers this step in depth.
7. Pick the fix that matches the source.
   • Your own page: correct the contradiction, and make sure the fact is in the HTML before JavaScript runs, since most AI crawlers do not execute JavaScript.
   • Your site is unreachable: check robots.txt per bot (OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot) and whether your firewall returns 403 to them.
   • A third-party page: ask the publisher to correct it. For hostile pages, see what to do about a competitor's comparison page.
   • Trained knowledge with no citations: nothing you ship this quarter changes it. Fix what search can reach and log the row as waiting on the next model.
8. Ship, then re-test. Mark the row shipped with a date. Re-run the same prompt on the same engine in the next cycle.
9. Verify or reopen. One clean answer is a good sign, not proof. Mark it verified after two consecutive clean re-tests, or reopen it.

If the same wrong fact appears across several engines, link those rows. A shared source is the usual explanation, as covered in why the same wrong fact shows up on several platforms.

The headline number should be errors divided by answers that returned. Do not add up penalty points and subtract from 100.

A raw penalty sum punishes you for measuring more. In 99Visibility's own scoring review (September 2026), a summed-penalty formula scored an audit of 16 answers with 2 minor errors at 90 and an audit of 200 answers with 20 minor errors at 0. Both had roughly a 10% error rate. The second team just asked more questions.

Established factuality metrics avoid this by dividing by a count. FActScore divides supported facts by atomic facts in the response. RAGAS faithfulness divides supported claims by total claims. Use the same idea:

• Error rate: answers with at least one wrong claim, divided by answers returned.
• Answers returned: exclude prompts where the platform errored or refused. The Vectara hallucination leaderboard publishes an answer rate for this reason, so refusals are not mistaken for accuracy.
• Worst severity: report the most severe open error next to the rate. The SAFE paper (Wei et al.) added a recall term because a precision-only rate can hide problems. In practice, one critical error can vanish inside an average of 400 answers.
• Per-engine rates: one blended rate hides an engine that is badly wrong.

Be honest about severity weights. FActScore treats every fact as equal weight. Weighting critical above minor is a reasonable business choice, not a benchmark standard, so label it as yours. For more on why a sampled count is not the full picture, compare prompt-based tracking with engine-reported data.

Good

• The error rate on a fixed prompt set falls across cycles, engine by engine.
• Critical errors reach zero open rows and stay there through re-tests.
• Every open row has a suspected source and an owner.
• Errors traced to your own site close fastest, because you control the fix.
• Rows tagged trained knowledge are few, and you report them as outside your control.

Bad

• The rate moves because the prompts changed, not because the answers did.
• A rising penalty total gets reported as falling accuracy after you expanded the prompt set.
• Rows sit open with "unknown source" for months.
• A fix is marked done after one clean answer.
• Retrieval state is guessed instead of recorded from citations.

No tracker promises an engine will stop repeating an error. What it shows is which fixes you shipped and whether the answers changed after them.

Set a rhythm and keep it.

• Monthly: re-run the full prompt set on every engine you track and recompute the rates.
• Weekly for critical rows: re-test open critical errors until they verify.
• After you change a key page: re-test the prompts that touch that fact in the next cycle.
• After a model launch or default change: re-run everything. Engines change the models they serve, and an error can return or disappear for reasons unrelated to your site.

Each run is a snapshot. Date every row so a reader can see when each answer was true. To get a first baseline before you build anything, the free checker on this site runs up to three buyer questions and scores the answers. A narrower first pass is the free AI pricing accuracy check.

• A fact sheet of true values, each linked to the page that states it
• A fixed prompt set, saved and versioned
• Logged-out or memory-off sessions for every run
• Separate engine values for AI Overviews, AI Mode and the Gemini app
• Retrieval state recorded from visible citations only
• Error type that splits contradicted facts from unverifiable ones
• Severity scale written down, and labelled as your own choice
• Error rate computed over answers returned, with refusals excluded
• Worst open severity shown next to the rate
• A per-engine breakdown in every report
• An owner and status on every open row
• A re-test date on every shipped fix

When the log points to your own site as the source, the fix plan for AI getting your brand wrong lays out the order of work.

How do I know if a wrong answer came from live search or the model's training? Look for citations. If the answer shows links, it used search, and those links are your first suspects. If it shows none, record the retrieval state as unknown. The answer may come from trained knowledge with a fixed cutoff, and that cutoff varies by model even inside one product. For ChatGPT specifically, see [trained knowledge versus search errors](/platforms/correct-chatgpt-brand-facts).

Should I log claims I cannot verify as errors? Log them, but in a separate type. An engine stating a number you never published is a gap, not proof of a falsehood. Treat it as a prompt to publish the true value plainly on a page crawlers can fetch. Keep unverifiable rows out of the main error rate, or report them as a second rate, so your headline number reflects claims that contradict your records.

Will adding schema markup correct a wrong AI fact? There is no evidence it does. Google says structured data is not required for its generative AI features, and Ahrefs' matched test found AI Overview citations fell 4.6% after pages added schema. Existing markup does no harm. The fixes with evidence behind them are access fixes and stating the correct fact clearly in HTML that loads without JavaScript.

Why does my error rate change when I didn't change anything? Engines change models, defaults and retrieval behavior on their own schedule. Search also fires per query, so the same prompt can return a searched answer one day and an unsearched one the next. Keep your prompt set fixed, run logged out, and date every row. If a rate shifts with no site change, check for a model or default change before drawing conclusions.