What should an AI visibility tool disclose about its methodology?
99Visibility Research · Updated October 21, 2026 · 5 min read
Short answer
An AI visibility tool should disclose five things. First, which model it calls on each platform and how that differs from what the consumer app serves. Then whether web retrieval was on or off, the exact prompts it ran, how many answers each number rests on, and its score formula with a version date. Without those, you can't tell a real change in AI answers from a change in the tool.
Key takeaways
- A tool that calls an API model is measuring a proxy for the consumer app, and it should say which proxy it uses.
- Retrieval on or off changes the answer more than almost anything else, so every result needs its retrieval state labelled.
- Prompts are a sample of an input nobody outside the engine can see, so the prompt list and answer count belong next to every score.
- A score formula change breaks the trend line, so scores need a formula version and old and new scores should not be compared.
- Ask for raw answers, not only a number, before you trust or act on any AI visibility score.
The five disclosures, and why each one matters
Every AI visibility number is the output of choices the tool made before you saw it. A buyer should be able to see those choices. Here is the short list.
| Disclosure | Why it matters |
|---|---|
| Model called per platform, versus what the app serves | The API model may differ from the consumer app's default, with a different knowledge cutoff |
| Retrieval on or off | An answer with live web search and one from training data alone are different measurements |
| The prompts used | Results depend on the questions asked, and you need to judge whether they match your buyers |
| Sample size per number | Small samples swing, and a per-platform figure from a handful of answers can mislead |
| Score formula and its version | A formula change moves the score with no change in what AI says about you |
The reason is simple: a score you can't decompose can't guide a decision. If you want the background on what scores average together, read what an AI visibility score averages. If two tools already gave you different numbers, these five items usually explain why AI visibility tools disagree.
Models and retrieval state
"ChatGPT" is not one thing. OpenAI's model documentation lists a February 16, 2026 knowledge cutoff for its GPT-5.6 family and an August 31, 2025 cutoff for the Instant path used in ChatGPT. A tool that calls one model and labels the result "ChatGPT" is measuring one of several possible states.
Google is the same. Google says AI Overviews and AI Mode "may use different models and techniques", and returns different links. Calling the Gemini API with search grounding is a proxy for the Gemini app. It is not AI Overviews.
Retrieval matters even more. As of October 2026:
- ChatGPT decides per query whether to search.
- Web search in Claude's consumer app is a user toggle with no published default.
- Perplexity's Agent API requires web search to be switched on.
So a tool should state, per platform: the model it called, whether search was on, and whether answers were logged out. OpenAI says ChatGPT may use saved memories when rewriting a search query, so logged-in results aren't reproducible. If a tool reports one blended number without this, it's averaging answers given with and without live retrieval.
Prompts and sample size
Engines don't search with your prompt. OpenAI says ChatGPT search "typically rewrites your query into one or more targeted queries". Google may fan a query out across subtopics. That means any prompt set is a proxy for queries nobody outside the engine can see. It doesn't make prompts useless. It means you need to see them.
Ask for three things:
- The full prompt list, word for word.
- How the prompts were chosen (written by you, generated from your site, or picked by the vendor).
- The number of answers behind each figure, per platform.
Sample size has a second trap. A formula that subtracts penalty points without dividing by the number of answers will push your score down when you buy more prompts, even if the error rate stays flat. A sound formula normalises by answers returned, not answers requested. Whether more prompts should change your score covers this in detail.
Score formulas and version changes
Every composite score has weights, and the weights are product decisions. Good tools say so. Factuality research like the SAFE paper shows why design matters: a pure average can let one severe error fade toward nothing as the sample grows, so a formula needs a way to keep the worst error visible.
What a buyer should expect:
- The components and their weights, in plain words.
- Which constants were chosen by judgement rather than derived from data.
- A version label on every stored score, and a dated changelog.
- A clear statement that scores from before a formula change aren't comparable with scores after it.
Without versioning, a formula update looks like a gain or loss in AI visibility. That's the most common way a trend line lies. Before you change anything on your site, set a baseline and note the formula version it was scored with.
What no tool can disclose, and what to ask next
Some things can't be measured by anyone outside the engines. Be wary of a tool that claims otherwise.
- Rank or position in an AI answer. Google's Search Console report for its AI features exposes impressions only. No ground truth exists to check a position number against.
- Influence. OpenAI's API returns more consulted sources than cited ones, so a citation count undercounts what shaped the answer.
- Ranking factors. OpenAI says ChatGPT ranks results using "multiple factors" and lists none. See what OpenAI discloses about ChatGPT ranking.
The evidence base is thin in general. A July 2026 critical survey of 45 GEO studies found no technique with a stable, cross-platform causal effect on discoverability. A tool that's honest about this will label which numbers are measurements and which are its own model.
Your next move: ask any vendor for one audit's raw answers, with model, retrieval state, prompt and date on each. If they can't produce it, the score can't be checked. Seeing the full AI answers is the fastest test of a tool's methodology.
Frequently asked questions
Is a tool hiding something if it won't share its prompts?
Not necessarily, but you lose the ability to judge the result. The score depends on which questions were asked, so a prompt list that doesn't match how your buyers search can make you look better or worse than you are. Ask at least for the prompts behind your own audit. A tool that won't show those is asking you to trust a number you can't check.
Can I compare scores from two different AI visibility tools?
Usually not directly. Tools call different models, run different prompts, set retrieval differently and weight components differently. Two tools can both be measuring correctly and still disagree. Compare the raw answers instead: if both show ChatGPT stating a wrong price for you, that finding holds no matter which score each tool prints.
Should a tool measure logged in or logged out?
Logged out, with memory off, if it wants reproducible results. OpenAI says ChatGPT may use saved memories when rewriting a search query, so a logged-in answer reflects one user's history. A tool should state which mode it used. If it measures logged in, treat each result as one person's view, not a general one.
How often should a tool's methodology change?
As often as the engines change, which is frequently. Models and defaults in consumer apps shift every few months. What matters is that each change is dated and every stored score carries the methodology version that produced it. Frequent, documented changes are a good sign. Silent ones are the problem.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.