Should AI visibility tools test with web search on or off?

A good AI visibility test runs each prompt in both states and reports them separately. The two states answer different questions, so they deserve different columns.

• Search off tells you what the model believes about your brand from training data alone. This is the answer a user gets when the app decides not to search.
• Search on tells you whether the engine can find, fetch and cite your pages right now. This is the answer a user gets when the app does search.

The problem is the consumer apps. They don't announce which state they used. If a tool tests in one state and calls the result "ChatGPT visibility", it measures half the picture. If it tests in whatever state the app picks, it averages an unknown mix. Either way, a single number hides the thing you need to know: which half is broken. For the precise definitions, see retrieval-on vs retrieval-off answers.

The evidence base separates AI answers into four stages: parametric knowledge, crawler access, retrieval and generation. Search off touches only the first. Search on touches all four.

Cutoffs make the search-off column concrete. OpenAI's model docs list a February 16, 2026 knowledge cutoff for the GPT-5.6 family and August 31, 2025 for the Instant path behind chat-latest. So "ChatGPT's knowledge" is not one value, and a launch from last spring may simply not exist in the ungrounded answer. That is a fact to record, not a problem you can fix by editing your site.

Search-on failures are different. They often trace back to access: a robots.txt rule, a firewall returning 403, or content that only appears after JavaScript runs. Those are in your control, which is why the search-on column is the one that produces a to-do list.

Retrieval is discretionary on every major platform, as of October 2026.

• ChatGPT decides per query whether to search. OpenAI's help page says ChatGPT search "typically rewrites your query into one or more targeted queries", and that saved memories can shape that rewrite when memory is on.
• Claude.ai treats web search as a user toggle, with no published default.
• Perplexity's Agent API requires developers to opt web search in.
• Google AI Overviews often do not trigger at all, and Google says AI Overviews and AI Mode "may use different models and techniques".

Nobody agrees on how often ChatGPT searches. Semrush's US clickstream put it at 34.5% of queries in February 2026, Profound reported about 18% of conversations, and cloro reported 80.5% globally on a 100-prompt set. These measure different things on different populations, so none of them tells you the mix for your buyer questions. If you see an answer without citations, you usually can't tell which state produced it. This page covers how to tell whether an answer used live web search.

Silent loss of grounding

Perplexity retired Sonar Chat Completions on September 27, 2026. Code that was not migrated to the Agent API can return ungrounded answers without any error. A tracker in that state reports search-off results while labelling them as Perplexity search. The Sonar to Agent API migration docs explain what changed.

Retrieved is not the same as cited

OpenAI's API returns a sources list separate from citations, and OpenAI says the number of sources "is often greater than the number of citations". A search-on test that counts only citations misses pages the model read but didn't credit. You can log retrieved and cited sources separately.

The API model is not always the app model

The model a tool calls may not be the one a free or paid user gets in the app. Check which ChatGPT version a tool tests before comparing scores across tools.

Logged-in state

Memory and custom instructions change query rewriting. Test logged out or memory off, or don't claim the results are reproducible.

Set up your testing so every number carries its retrieval state.

1. Pick your buyer questions and run each one through the API twice: search forced on, search forced off.
2. Store the state, model name, date and full answer with every result.
3. Compare the two answers for the same prompt. Wrong facts in both point to old training data. Wrong facts only in the search-on answer point to the pages being retrieved, which may be yours or a third party's.
4. For search-on failures, check access first: does robots.txt allow OAI-SearchBot, PerplexityBot and Claude-SearchBot, does your CDN let them through, and is your key content in the HTML before JavaScript runs.
5. When a vendor shows you one blended score, ask which retrieval state each answer came from.

The step-by-step method is in testing AI answers with search on and off. For a quick look at your own brand, the free search on vs off test shows how the two answers differ. No test controls what a model says. It tells you which part of the pipeline to fix.

Is a search-off test useless if I can't change training data? No. It tells you what a large share of users may hear when the app skips search, and it dates your brand in the model's memory. If the ungrounded answer has your old pricing, you know the problem is the snapshot, not your site. That stops you from chasing on-page fixes for an answer no page edit can reach until a newer model ships.

Does search on mean the AI will find my pages? No. Search on only means the engine looked. Your page still has to be reachable by that engine's crawler, present in its index, and selected for the query. Allowing a crawler buys eligibility, not retrieval or citation. Many search-on misses trace back to blocked bots, firewall rules, or content that only loads with JavaScript.

Can I test retrieval state in the consumer apps directly? Only partly. Claude.ai lets you toggle search, but ChatGPT decides per query and Google AI Overviews may not trigger at all. The API gives you control: you choose whether the web search tool is available. That makes API testing the reproducible option, with the caveat that the API model may differ from the one an app user gets.

Why do my search-on and search-off answers disagree? They draw on different evidence. Search off relies on what the model learned before its cutoff. Search on adds pages fetched today, after the model rewrites your prompt into its own queries. Disagreement is useful: it shows whether a wrong fact lives in the model's memory, in retrieved pages, or in both.