Testing AI Answers With Search On and Off

An AI answer about your brand comes from one of two places. With search off, the model answers from parametric knowledge: what it absorbed in training, frozen at a cutoff date. With search on, the engine fetches pages first and writes from them, a process called grounding.

The cutoff is a hard ceiling. As of October 2026, OpenAI's model documentation lists a February 16, 2026 knowledge cutoff for the GPT-5.6 family and August 31, 2025 for the Instant path (chat-latest). A price change you shipped in March 2026 cannot appear in a search-off GPT-5.6 answer. Nothing you do this quarter changes that.

The search-on answer is different. It depends on whether a crawler could fetch your page, whether the engine retrieved it for that query, and which retrieved pages the model chose to use. The first of those is yours to fix.

Consumer apps blend both states. ChatGPT searches at the model's discretion, and Claude.ai search is a user toggle with no published default. So a single app test measures an unknown mix. The fix is to set the retrieval state yourself, through the API, and test each state on purpose. For the longer explanation, see why search-on and search-off answers differ.

Get these right first. A sloppy setup produces differences that come from your test, not from the engine.

• Pick 5 to 10 fact questions with one checkable answer each: starting price, founding year, headquarters city, current product names, key integrations, ownership.
• Write down the correct answer for each, with the URL on your own site where it is stated.
• Confirm each fact is in the raw HTML. View source, not the rendered page. If the price only appears after JavaScript runs, most AI crawlers will not see it.
• Use the API, not the logged-in app. Saved memories and custom instructions change how ChatGPT rewrites a query, which is why logged-in and logged-out tests differ.
• Pin and record the exact model name and the date of every run.
• Keep prompt wording and sampling settings identical across both states. Only the search setting should change.
• Run each prompt several times per state. We suggest three. Answers vary between runs, and one sample can mislead.
• Store the full response, including every source and citation field, not just the answer text.

Run each question through each engine twice: once with no search tool, once with it enabled.

1. OpenAI. Send the prompt to the Responses API with no tools. Then send it again with the web_search tool enabled. Save the retrieved sources and the citations as separate lists. OpenAI's documentation notes that the number of sources is often greater than the number of citations.
2. Anthropic. Call the Messages API without and then with the web search tool. Anthropic's docs say results expose url, title and page_age, and citations quote a short cited_text span. As of October 2026 the docs list web search at $10 per 1,000 searches plus tokens.
3. Perplexity. Use the Agent API, where web search is opt-in. Sonar Chat Completions retired on September 27, 2026, and code that was never migrated can return ungrounded answers without any error. Check that search_results is actually populated.
4. Gemini. Call the Gemini API without and then with Grounding with Google Search. Google's docs say the model generates and runs its own search queries. Note that this is not Google AI Overviews or AI Mode, which Google says may use different models and techniques.
5. Verify the retrieval state of every search-on response. If it came back with no sources, log it as a search-off answer. Here's how to tell whether an answer used live web search.
6. Record, for each search-on run, whether any URL from your domain appears in the sources, and separately whether it appears in the citations.

Compare facts, not wording. Two answers can read differently and state the same price. Label every fact in every answer with one of five outcomes:

• Correct: matches your written answer.
• Outdated: was true once, for example a retired plan name or an old price.
• Wrong: never true, or describes another company.
• Missing: the answer skipped the fact or declined to give it.
• Unverifiable: a specific claim you have no record to check against.

Then put the counts side by side, per engine and per state.

The last two columns matter most. They tell you whether a wrong search-on fact came from your page, from someone else's page, or from no page at all.

A healthy result is not a perfect search-off answer. It is a search-on answer that gets your facts right and pulls them from pages you control.

Good

• Search-on answers are correct, and your URL is among the retrieved sources.
• Search-off answers are outdated only in ways that match the model's cutoff. That is expected and not a failure.

Bad, and what each pattern points to

• Search-on is wrong and search-off is right. The engine retrieved a page with a stale or wrong fact. Check your own pages for contradictions first, then the third-party pages in the source list.
• Search-on retrieves nothing from your domain for a question only you can answer, such as your own pricing. That points to an access problem. Check crawlers before anything else.
• Search-on takes your fact from a third-party site. Being named accurately on the sites engines already cite is associated with appearing in the answer, though no study promises an effect.
• Search-off states something that was never true. That is a hallucination in training-era knowledge. You cannot correct it this quarter. Your realistic goal is a correct search-on answer.
• Your URL is retrieved but not cited. Citation is a noisy signal of what the model used, not proof. Log it and watch the trend.

Web search does not guarantee accuracy either. Read more on whether web search makes brand answers more accurate.

Search-on failures usually trace back to access. These are the checks with evidence behind them.

• robots.txt, bot by bot. OAI-SearchBot controls ChatGPT search. GPTBot controls training only, and OpenAI says each setting is independent of the others. Also check PerplexityBot and Claude-SearchBot.
• Firewall and CDN. A bot-management rule can return 403 to a crawler that robots.txt allows. From September 15, 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers, so access can change without anyone editing your site.
• Rendering. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Put the price, the product names and the one-line definition in the server-rendered HTML.
• One stated fact, everywhere. If your pricing page, footer and about page disagree, the engine can quote any of them. Make them match.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Skip schema, FAQ markup and llms.txt as fixes for this problem. No credible evidence shows they get pages cited, and fixing access removes a block rather than promising a citation.

The two states change on different clocks, so schedule them separately.

• Search-on: run a baseline now. Rerun within a few days of any access fix, such as a robots.txt edit, a CDN rule change or a rendering deploy. Otherwise, rerun monthly.
• Search-off: rerun only when the model changes. A frozen model gives a frozen answer. OpenAI warns that model snapshots are regularly updated, and defaults shift, so watch release notes for the models you test.
• After a pricing or product change: rerun the search-on questions that cover the changed fact, and expect search-off to stay outdated until a model with a later cutoff ships.

Keep the prompts, models and settings fixed across runs so results stay comparable. A reproducible prompt testing protocol covers how to log each run so next month's numbers mean something.