How to Choose an AI Visibility Tracking Tool: Five Checks That Matter

An AI visibility tracking tool sends a set of prompts to AI engines, records the answers, and reports whether and how your brand appears. That is the whole mechanism. Our definition of AI visibility tracking covers the term in more detail.

The answer you see is the end of a four-stage chain:

• Parametric: what the model knows with no web access, frozen at its training cutoff.
• Access: whether a crawler can fetch and store your page at all.
• Retrieval: whether your page enters the model's context for a given query.
• Generation: which retrieved pages the model uses and cites.

A tracker only sees the final output. So the best tools do two jobs: they record what each engine said, and they help you work out which stage caused the result. Use that as the frame for every criterion below.

Engines behave differently, and a single number averages that away. Google's documentation says AI Overviews and AI Mode "may use different models and techniques" and can return different links. Citation units also differ: Anthropic cites short text spans, while Perplexity cites whole sources by index. There is no shared denominator to average across.

The gap can be large. In Otterly's 2026 experiment, 2,718 of the 3,934 citations it tracked for one test site came from Microsoft Copilot, while ChatGPT, Gemini, AI Overviews and AI Mode almost never cited it. A blended score would have hidden all of that.

What to ask a vendor

• Does the dashboard show each engine on its own line?
• Are Google AI Overviews, AI Mode and the Gemini app reported as separate surfaces?
• If there is a combined score, is it labelled as the vendor's own model rather than a measurement?

Retrieval is decided per query, and the rules differ by engine. As of October 2026, ChatGPT search runs at the model's discretion, Claude.ai search is a user toggle with no published default, and web search on Perplexity's Agent API must be switched on by the developer. An answer generated with search off reflects only what the model learned before its cutoff. OpenAI documents different knowledge cutoffs for different ChatGPT model paths, so "what ChatGPT knows" is not one value.

This matters for interpretation. A brand that launched a product last month can look invisible in an ungrounded answer and present in a grounded one. Neither result is wrong, but they mean different things and call for different fixes.

The prompts are also not what the engine searches for. OpenAI says ChatGPT search "typically rewrites your query into one or more targeted queries" before sending them to search providers.

What to ask a vendor

• For each engine, was web search on or off, and which model answered?
• Were answers collected logged out with memory off, so results are reproducible?
• Does the tool record retrieved sources separately from cited ones? OpenAI states the number of sources consulted is often greater than the number cited.

Accuracy: is what AI says true?

Being mentioned is only half the story. AI platforms get specific facts wrong about specific brands: prices, product names, ownership, features. A useful tool compares answers against facts you supply and flags the wrong ones by severity.

Check how the accuracy number is built. Established factuality metrics such as FActScore divide errors by a count of facts or answers. A score that simply subtracts penalty points can fall when you add more prompts, even if the error rate stays flat. Ask whether failed or empty answers are excluded from the count, and whether one serious error stays visible in a large sample.

Access: can AI crawlers reach you?

Tracking shows the outcome. Access checks show causes you can fix, and they carry the strongest evidence in the field.

• robots.txt is per bot. OpenAI's crawler page says GPTBot governs training, OAI-SearchBot governs ChatGPT search, and each setting is independent.
• Firewalls can override robots.txt. OpenAI's help centre says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses.
• JavaScript can hide content. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so client-side rendered content can be invisible to them.
• Defaults change without you. From 2026-09-15, Cloudflare blocks training and agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers.

Ask whether the tool requests your pages as each AI crawler and compares raw HTML with the rendered page. If a tool only tracks answers, pair it with something that diagnoses. Our comparison of tracking tools versus fixing tools explains the split.

Every AI visibility score is a model built by the vendor. That is fine, as long as the vendor shows its work. Ask for the formula, the weights, and the version history. Ask whether scores from before a formula change are comparable with scores after it. If the vendor can't answer, the trend line on your dashboard may not mean what it shows. Our playbook on choosing AI visibility KPIs you can defend goes deeper on which numbers hold up.

Red flags in a sales pitch

• An AI rank or position in Google AI. Google's generative AI report exposes impressions only, so there is no ground truth to check a rank against.
• An E-E-A-T score. Google says E-E-A-T "isn't a specific ranking factor". There is no score to measure.
• Domain authority as an AI lever. It is a third-party metric that no engine computes.
• "Add schema to get cited." Ahrefs' matched-control study found AI Overview citations fell 4.6% on pages after schema was added.
• "Publish an llms.txt." Google says Search ignores it, and no other engine documents reading it.
• "Up to 40% more visibility." That figure comes from a lab engine with sources already loaded into context.

A 2026 critical survey of 45 GEO studies found no technique with a stable, longitudinal, cross-platform causal effect on discoverability. A tool that promises one is overselling.

Be clear about the limits before you buy anything.

In your control:

• Crawler access: robots.txt per bot, firewall and CDN rules.
• Server-side rendering of the facts that matter, such as pricing and product names.
• Snippet controls like noindex, nosnippet and data-nosnippet, including ones applied by accident.
• Stating specific facts clearly on pages that can be fetched, and keeping them consistent across your site.
• Being named on the third-party pages engines cite instead of yours (an association in the evidence, not a guaranteed effect).

Outside anyone's control:

• What a model learned before its cutoff.
• Whether an engine decides to search for a given query.
• Where your page lands in the model's context, and which sources it cites.

No tracker sees the real queries engines run. Every prompt set is a sample. In Otterly's experiment, Microsoft's own AI Performance report counted 42,817 citations for a site where Otterly's prompt tracker logged 3,934, though the time windows may not match. Treat any tool's numbers as a sample, and say so when you report them.

Work through these steps before signing a contract.

1. Take a baseline. Run a snapshot so you know where you stand before any demo. The free checker on this site scores up to three buyer questions. A snapshot is not tracking, and our page on what a free snapshot misses explains the gap.
2. Write your must-have list using the table below.
3. Ask each vendor the same questions in writing. Compare answers, not demos.
4. Run a short trial on your real prompts. Check whether results are reproducible across two runs.
5. Check the price against coverage. Our guide to what AI visibility tools cost gives context without naming winners.

• Counting engines instead of checking methods. Five engines measured badly tell you less than three measured with disclosed retrieval state.
• Reading a trend across a formula change. If the vendor changed its scoring, older numbers may not be comparable.
• Treating mentions as traffic. A citation is not a visit, and Google's AI report shows no clicks.
• Over-weighting competitor prompts. Prompts that name rivals answer a different question. See whether to track competitor-name prompts.
• Switching tools without exporting history. Your baseline goes with the old account. Read what to check when switching AI visibility tools first.
• Buying a tracker to solve an access problem. If a firewall returns 403 to OAI-SearchBot, no amount of tracking will fix it. Check access first.