Free Test: AI Answers With Search On vs Off

Every AI answer about your brand comes from one of two places. The model either answers from what it learned in training, or it searches the web first and answers from the pages it fetched. Our glossary entry on retrieval-on and retrieval-off answers defines both.

The search-off answer is the model's parametric knowledge. It is frozen at a knowledge cutoff. OpenAI's model documentation lists a Feb 16, 2026 knowledge cutoff for its GPT-5.6 models (Sol, Terra and Luna). The Instant path, chat-latest, lists Aug 31, 2025. So "ChatGPT" is not one knowledge state. The cutoff depends on which model answers.

The search-on answer is built from pages fetched at answer time. That is the answer your site can affect.

Comparing the two tells you three things:

• What the model believes about you with no help from your site.
• Whether a web search corrects or repeats that belief.
• Which pages the engine actually cites when it does search.

You need a free account on one or two AI apps and about 20 minutes.

1. Write three buyer questions. Use the questions a prospect would ask. Say you run a dental clinic in Austin: "What does a cleaning cost at [clinic]?", "Is [clinic] open on Saturdays?" and "What are the best dental clinics in Austin?"
2. Start a clean session. Log out, or use a temporary chat with memory turned off. OpenAI says that if memory is enabled, ChatGPT may use saved memories when it rewrites a search query. That makes your result personal, not representative.
3. Run the search-on version. In ChatGPT, ask "Search the web: [question]", or turn on web search from the composer if your app shows the option. In Claude.ai, web search is a user toggle, so switch it on first. Save the answer and every cited URL.
4. Run the search-off version. Open a new chat. In Claude.ai, switch web search off. In ChatGPT, ask "Answer only from what you already know. Do not search the web: [question]." Check that no citations appear. If they do, the engine searched anyway, so discard that run.
5. Record the details. Write down the date, the app, the plan, the model name if shown, the cited URLs and each fact the answer states (price, hours, location, product names).
6. Repeat each pair two or three times. Answers vary between runs. One pair is an anecdote. Three pairs show a pattern.

Put the two answers side by side and check each fact against your own site. Most results fall into one of four patterns.

The cited URLs matter as much as the answer. If the engine cites a directory, a review site or a competitor instead of you, those pages are shaping the answer. Being named accurately on those sites is associated with appearing in the answer, though no study promises an effect. If you're unsure whether a given answer searched at all, see how to tell if an AI answer used live web search.

The search-off answer is out of reach this quarter. It changes only when the vendor ships a new model, and nobody has measured whether allowing training crawlers changes what a model says about a brand.

The search-on answer depends on whether the engine can reach and read your pages. These are the things to check:

• robots.txt, per bot. OpenAI says its settings are independent of each other. GPTBot covers training. OAI-SearchBot controls whether you appear in ChatGPT search. Check PerplexityBot and Claude-SearchBot too.
• Firewall and CDN. A firewall can return 403 to a crawler that robots.txt allows. OpenAI's help page says eligibility needs both the robots.txt permission and a host or CDN that allows its published searchbot IP addresses.
• JavaScript rendering. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Put prices and key facts in the HTML the server sends.
• Snippet controls. noindex, nosnippet, max-snippet:0 and data-nosnippet keep content out of AI answers. Make sure none were applied by accident.
• Plain facts. State the price, the hours or the one-line definition on a page that can be fetched. That gives the engine a sentence it can quote.

For ChatGPT specifically, our page on fixing ChatGPT errors from trained knowledge vs search walks through each case.

A hand test is useful, but it is a snapshot with known gaps.

• Search is discretionary. As of October 2026, ChatGPT decides per query whether to search. You can ask for it, but you can't guarantee what a real user's session does.
• Your prompt isn't the real query. OpenAI says ChatGPT search typically rewrites your question into one or more targeted queries. You never see those queries.
• Cited is not retrieved. OpenAI's API docs say the number of sources consulted is often greater than the number cited. The consumer app shows only citations. Our playbook on logging retrieved vs cited sources covers the API method.
• Sessions differ. Plans route to different models, and memory or custom instructions change results. See logged-in vs logged-out testing.
• Small samples. Three questions and a few runs show a pattern, not a rate.

Treat each result as dated evidence. Rerun it after you change your site and after a major model release.

If you want a quicker first pass before you open the apps, try the free AI Search Simulator, then confirm what matters with the manual steps above. The free checker on this site is another snapshot: it runs up to three buyer questions and scores the answers.

If your search-off answer carries old news, the free knowledge cutoff check helps you see which of your announcements came after a model's cutoff.

Does ChatGPT always search the web before answering? No. As of October 2026, ChatGPT decides per query whether to search, so some answers come from trained knowledge only. Answers with no citations are usually search-off answers. You can ask it to search, but you can't control what happens in a real user's session. That's why a fair test runs both versions on purpose and records which one produced each answer.

Why does the search-off answer mention old prices or products? The model learned about you before its knowledge cutoff and hasn't seen anything since. OpenAI lists Feb 16, 2026 for its GPT-5.6 models and Aug 31, 2025 for the Instant path. Anything you changed after those dates can't appear in a search-off answer. It can appear only when the engine searches and fetches your current pages.

Will allowing GPTBot fix what ChatGPT says without search? Not in any way anyone has measured. GPTBot governs training use only, and OpenAI makes no claim that allowing it improves search visibility. Any training effect would arrive with a future model generation. For ChatGPT search, the bot that matters is OAI-SearchBot. OpenAI says each crawler setting is independent of the others.

How often should I rerun the search on vs off test? Rerun it after you change something on your site, such as fixing rendering or a firewall rule, and after a vendor announces a new model. Keep the same questions and the same clean-session setup each time. Comparing dated snapshots shows whether a fix changed the search-on answer. It won't change the search-off answer until the model itself changes.

Can I choose which answer my customers see? No. The engine decides whether to search, and the user's app, plan, memory and settings also play a part. What you control is whether your pages can be fetched and read when the engine does search. Clean access, server-rendered facts and correct snippet settings make the search-on answer possible. They don't guarantee it.