Setting an AI Visibility Baseline Before You Ship Changes

A baseline is a snapshot you can compare against later. It records what AI engines could see, what they said, and what they got wrong, on a known date.

Most teams skip it. They ship a fix, check ChatGPT a week later, and read the result as proof. That reading fails because AI answers move on their own. Retrieval is per query and discretionary on every major platform, as of October 2026. OpenAI says ChatGPT search rewrites your question into one or more targeted queries before it searches, per its help article on searching the web. The same question can produce different answers on different days.

A useful baseline separates the four stages that shape an answer:

• Parametric: what the model knows with no web access. Nobody changes this quickly.
• Access: whether crawlers can fetch and read your pages. You control this.
• Retrieval: whether your page enters the answer's sources for a query. You control this only partly.
• Generation: which retrieved sources the model uses and cites. The model controls this.

Start with access. If a crawler is blocked, every mention count after that is measuring the block. That's why it pays to fix crawler access before tracking in earnest.

Do all of this on one day, before the fix ships. Write the date at the top of the record.

1. Fix the question set. Write the buyer questions you care about and freeze the wording. Phrasing changes answers, so how someone phrases a question matters. Save the exact text.
2. Fix the test conditions. Note each engine, the plan you used, and whether you were logged out with memory off. OpenAI says ChatGPT may use saved memories when rewriting a search query, so logged-in results are not reproducible.
3. Record robots.txt per bot. Rules are per crawler. OpenAI's crawler page says GPTBot governs training and OAI-SearchBot governs ChatGPT search, and "each setting is independent of the others." Record allow or disallow for OAI-SearchBot, GPTBot, PerplexityBot, Claude-SearchBot, ClaudeBot, Meta-WebIndexer, Googlebot and bingbot.
4. Record what the firewall actually returns. A CDN can return 403 to a crawler that robots.txt allows. Request a key page as each crawler and record the status code. The free AI Crawler Checker does this request for you.
5. Record what's in the raw HTML. Vercel found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. List the key facts (price, product names, one-line definition) and note which appear only after JavaScript runs. This playbook shows how to find facts hidden behind JavaScript.
6. Record suppression and index state. Note any noindex, nosnippet, max-snippet:0 or data-nosnippet on key pages. Google says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. Note Search Console and Bing index status for those pages.
7. Run each question on each engine, more than once. For every answer, record: brand named or not, your URL cited or not, competitors named, and whether the answer showed citations at all. No visible citations means you can't tell whether it searched.
8. Flag accuracy errors. Compare each answer against your verified facts. Mark wrong prices, old product names and wrong ownership, with a severity for each.
9. Export first-party data. Save Search Console's generative AI report (impressions only) and Bing Webmaster Tools' AI Performance export, including grounding queries. Set up Bing Webmaster Tools first if you haven't.
10. Store it read-only. Save raw answers, screenshots and the tool version together. Don't edit the record later.

A baseline without a change log is a photo with no date on the next one. Keep a simple table and add a row whenever anything moves.

Log your own deploys to the day, ideally with the time. Log robots.txt edits, CDN rule changes, template changes and content updates separately, because each can move results for a different reason.

Log outside events too. Model launches, default model changes, API retirements and vendor policy changes all move answers without touching your site. Perplexity's Sonar Chat Completions retired on 2026-09-27, for example, which changed what some tracking code measured. Use crawler access alerts so a regression you didn't make still gets a row.

A before-after comparison shows that something changed. It does not show that your fix caused it. Several things move at once:

• The models change. Vendors update models and routing. ChatGPT's reasoning and Instant paths carry different knowledge cutoffs (Feb 16, 2026 and Aug 31, 2025, per OpenAI's model docs).
• Retrieval switches on and off per query. Any average mixes answers given with and without live search.
• Competitors change their sites. Their fix can lift or lower you.
• Your sample is small. A few questions run a few times is noisy.

The research reflects this. A 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found that no reviewed technique shows a stable, longitudinal, cross-platform causal effect on discoverability. Studies that do get closer to cause use controls. Ahrefs' schema study compared 1,885 pages that added schema against matched control pages that didn't.

You can borrow that idea cheaply:

• Hold out comparable pages or questions that the fix doesn't touch, and track them alongside.
• Re-run several times before and after, not once.
• Prefer mechanism checks. "OAI-SearchBot now gets a 200" or "the price is now in raw HTML" is a verified fix. A rise in mentions afterward is an association, and you should report it that way.

Tick every line before the fix ships.

• Date and time of capture written at the top
• Frozen question list, exact wording saved
• Engines, plans and logged-out, memory-off conditions recorded
• robots.txt allow or disallow recorded per bot, training and search bots separated
• Status code recorded for each crawler requesting a key page
• Key facts checked in raw HTML versus rendered page
• noindex, nosnippet, max-snippet and data-nosnippet checked on key pages
• Search Console and Bing index status for key pages saved
• Each question run on each engine more than once, raw answers saved
• Per answer: named, cited, competitors, citations visible or not
• Accuracy flags listed with severity
• Search Console AI impressions and Bing AI Performance exports saved
• Tool name and methodology version recorded
• Deploy log started, with the planned ship date
• Hold-out pages or questions chosen

A good baseline

• It's dated, per engine and reproducible by someone else from the saved conditions.
• It keeps raw answers, not just a score.
• It records access state, so a later drop can be traced to a block or ruled out.
• It names the tool's method. Scoring formulas change, and a formula change can move a score with nothing changing on your site. Ask what a tool should disclose about its methodology.

A bad baseline

• One blended score across all engines. Google says AI Overviews and AI Mode may use different models and techniques, so a blend hides where the change happened.
• Logged-in, single runs, or a question list that drifted.
• No date, or no deploy log.

Reading the after-results

Good: access checks that failed now pass, and key facts now appear in raw HTML. That's a fix you can prove. Mention or accuracy changes that also show up on hold-out questions point to an outside cause, not your fix. Bad: claiming a lift from one re-run, or ignoring an outside event that landed the same week.

Repeat on three triggers.

• Before every meaningful deploy. Capture a fresh baseline so each change has its own before.
• On a fixed schedule. Run the same frozen questions under the same conditions, at the same cadence. Read what changes between AI visibility checks to pick a cadence you can keep.
• After outside events. A model launch, a CDN policy change or a new tool method all justify a new baseline. Mark old and new records so nobody compares across the break.

Access checks are cheap, so run them more often than answer checks. For a quick first snapshot, the free checker on this site shows whether engines name you for up to three buyer questions. Treat it as one dated data point, not tracking.

How many buyer questions should a baseline include? Enough to cover the decisions your buyers actually make, and few enough that you can run each one more than once on every engine. Evidence doesn't set a number. What matters more is freezing the wording and keeping it identical across runs. A short list run consistently gives you a cleaner comparison than a long list that changes every month.

Can I compare my baseline against numbers from a different tool? Not reliably. Tools choose different questions, engines, retrieval settings and scoring formulas. Even one tool can change its formula, which makes old and new scores incomparable. Compare raw records captured the same way. If you must switch tools, run both side by side for a period and start a new baseline with the new tool's method recorded.

Does a higher mention count after a fix prove the fix worked? No. It shows an association. Models update, retrieval switches on per query, and competitors change their sites in the same window. Stronger evidence comes from repeated runs, hold-out questions the fix didn't touch, and mechanism checks, such as a crawler that now receives a 200 instead of a 403. Report the lift, and say what else changed.

Should I include Google Search Console AI data in the baseline? Yes, as one input. As of October 2026, Search Console's generative AI report shows impressions for AI Overviews and AI Mode, with no clicks or position. Export it on baseline day so you have the before. Bing Webmaster Tools' AI Performance report adds citations and grounding queries, which are the searches the AI actually ran.