A Reproducible Protocol for Testing AI Prompts About Your Brand

Ask ChatGPT the same question twice and you can get two different answers. That is normal behavior, not a bug. Our page on answer variability covers the mechanics. Three causes matter most for brand testing.

• Sampling. The model generates text with some randomness, so wording, order and which brands get named can shift between runs.
• Model routing. "ChatGPT" is not one model. OpenAI's model documentation lists a February 16, 2026 knowledge cutoff for the GPT-5.6 family and August 31, 2025 for the Instant path. A request routed to one or the other can know different things about you.
• Retrieval is discretionary. As of October 2026, ChatGPT decides per query whether to search the web. Published estimates of how often it searches range from about 18% (Profound) to 80.5% (cloro), because they measure different things on different populations.

Google adds another layer. Its documentation says AI Overviews and AI Mode "may use different models and techniques", so the two surfaces can cite different pages for the same question.

The practical meaning: a screenshot of one answer, good or bad, tells you what happened once. Decisions need a pattern.

Before you run anything, lock these conditions. Use this as a checklist every time.

• Logged out, or a clean account. Personal history and settings change answers. See why logged-in and logged-out testing differ.
• Memory off and custom instructions empty. OpenAI states that if memory is enabled, ChatGPT may use saved memories when rewriting a search query, and that product results consider memory and custom instructions. More on how memory affects brand answers.
• Fixed wording. Store each prompt as exact text. Paste it, never retype it.
• One prompt per fresh chat. Earlier turns in a conversation steer later answers.
• A fixed number of runs per prompt. Pick a number and keep it. Five runs per prompt per surface is a workable starting point.
• Same location and language. AI Overviews do not appear on every results page and vary by location and language.
• Record the date on every run. Models and defaults change without notice.
• Record the model if the interface shows it. If it does not, write "not shown". Never guess.
• Record whether citations appeared, and copy every cited URL.
• Keep search state separate. If a surface lets you toggle web search, test on and off as separate conditions, never mixed.

Say you run a hypothetical project management tool for agencies. Here is how you would test it.

1. Write the question set. Pick 10 to 20 questions a buyer would actually ask, such as "best project management tool for small agencies". Include category questions, comparison questions and direct questions about your brand ("how much does [brand] cost").
2. Freeze the wording. Save the questions in a spreadsheet with an ID each. Phrasing changes results, as covered in how phrasing affects brand mentions, so a reworded question is a new prompt with a new ID.
3. Write down the facts you expect. List your correct price, plans, founding year and key features. You need this to judge accuracy, not just presence.
4. Set up each surface. Open a private browser window, log out or use a clean account, and confirm memory is off.
5. Run each prompt in a fresh chat, the set number of times. Do not regenerate inside the same chat. Open a new one for each run.
6. Log every run immediately using the fields in the next section.
7. Repeat for each surface. ChatGPT, Gemini, Claude, Perplexity and Google's AI features behave differently. Score them separately.
8. Summarize per prompt and per surface. Count runs where you were named, runs where facts were wrong, and runs with citations. Do not blend surfaces into one number.

If you want a structured version of this for one engine, the ChatGPT prompt tracking method goes deeper.

A run you cannot reconstruct later is wasted. Capture these fields.

Treat citations with care. OpenAI's documentation notes that the number of sources consulted is often greater than the number cited. Research on how models cite (arXiv:2606.28358) found citation output can be changed without changing the sources. A citation is evidence a page was used, but a missing citation does not prove a page was ignored.

Good results are stable and correct. Bad results are either absent, inconsistent or wrong.

Signs of a healthy result

• You are named in most runs for a prompt, not one run in five.
• The facts match your fact list across runs.
• When citations appear, they point to your own pages or to accurate third-party pages.

Signs of a problem

• Named only when search is off. The model knows you from training data but the live web path is not finding you. Check whether your robots.txt allows retrieval bots such as OAI-SearchBot and PerplexityBot, and whether your CDN or firewall lets them through.
• Named only when search is on, with old facts when off. Training data is stale. You cannot change that this quarter, but you can make sure current facts are fetchable.
• Wrong facts that repeat across runs. A consistent error is more serious than random noise. Check whether your own site states the fact in different ways on different pages, or loads it only through JavaScript.
• Competitors cited from third-party sites you are absent from. Being named on those sites is associated with appearing in answers, though no effect size is promised.

The search on and search off playbook explains how to split these cases cleanly.

Run the full protocol on a fixed schedule, monthly for most brands. Keep the prompt set and run count identical so each month compares with the last.

Also rerun it after events that change the inputs:

• A major model launch or a change in a surface's default model.
• A site change that touches rendering, pricing pages or robots.txt.
• A CDN or firewall change. Cloudflare set training and agent crawlers to blocked by default on ad-displaying pages for new and free-tier sites from September 15, 2026, so a site can lose AI access without anyone editing it.
• A product or pricing change, since you want to see how long old facts persist.

When you change the prompt set, start a new series. Mixing old and new prompts breaks the comparison. For a quick snapshot between full runs, the free checker on this site asks up to three buyer questions and scores the answers, though it is a snapshot, not continuous tracking.

Be plain with your team about the limits.

• No rank exists to measure. Google's AI report exposes impressions only, with no position data. Any "AI rank" figure has nothing to be checked against.
• You cannot see the real queries. OpenAI says ChatGPT search typically rewrites your question into one or more targeted queries. Google may fan out into related searches. Your prompt is a proxy for an input you do not observe.
• One blended score hides the story. A single visibility number averages answers given with and without live retrieval, across surfaces that work differently. Keep results per surface and per search state.
• The test diagnoses, it does not fix. It shows where you are missing or misdescribed. The fixes that have evidence are access fixes: server-rendered facts, per-bot robots.txt permission, reachable servers and no accidental snippet blocks. Nobody controls what a model finally says.