SEO for AI Models: Training Data vs Live Retrieval

An AI model can mention your brand in one of two ways. It can answer from what it learned in training, called parametric knowledge, which is frozen at a knowledge cutoff. Or it can run a live web search, fetch pages and answer from what it just read.

Those two routes pass through four stages:

• Parametric: what the model knows with no web access. Nobody changes this until the next model ships.
• Access: whether a crawler may fetch and store your page at all. You and your CDN control this.
• Retrieval: whether your page enters the model's context for a given question. The engine controls this, partly.
• Generation: which retrieved pages the model uses and cites. The model controls this.

Retrieval fires per question, at the engine's discretion. ChatGPT search decides when to search. Claude.ai search is a user toggle. Google AI Overviews often do not trigger at all. So the same buyer question can get a training-only answer one day and a search-backed answer the next.

The practical move: before you judge an answer about your brand, find out whether that answer used live web search. The fix depends on which route produced it.

Training crawlers collect content that may be used to build future models. OpenAI describes GPTBot this way: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Anthropic's ClaudeBot plays the same role for Claude. Meta-ExternalAgent is Meta's equivalent.

Google-Extended works a little differently. It is a robots.txt token, not a separate crawler, and it controls training and grounding for Gemini Apps and Vertex AI. Google states it does not affect a site's inclusion in Google Search and is not used as a ranking signal.

Here is the part most advice skips. OpenAI nowhere claims that allowing GPTBot improves your visibility. No study in the evidence tests whether blocking training crawlers makes models forget a brand. The honest label is unmeasured, not proven harmful and not proven harmless. If that is your worry, read whether blocking training crawlers erases brand recall.

Timing matters too. Any effect lands only when a new model is trained and released. As OpenAI's model docs listed them when checked in August 2026, the GPT-5.6 family carried a February 16, 2026 knowledge cutoff, while the Instant path behind chat-latest carried August 31, 2025. That is why AI can repeat your old news, as covered in knowledge cutoffs and your brand. Content you publish this month cannot appear in a training-only answer from any model already shipped.

What this means: treat training-crawler access as a licensing decision about your content, not a visibility lever.

Retrieval bots build the indexes that live AI answers draw from. This is where access decisions have a documented consequence.

• OAI-SearchBot: OpenAI states that "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• Claude-SearchBot: Anthropic uses it to improve Claude's search results, separate from ClaudeBot.
• PerplexityBot: Perplexity says it indexes for search and is not used to crawl content for AI foundation models.
• Meta-WebIndexer: Meta gives the only first-party statement tying access to citation: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."
• Googlebot: Google AI Overviews and AI Mode have no robots.txt token of their own. A page must be indexed and eligible to show in Google Search with a snippet.

A third group fetches pages when a user asks. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Index-time and answer-time controls are different things.

Allowing a retrieval bot makes you eligible. It does not make an engine pick you.

Platform documentation changes. This map reflects vendor pages as verified in 2026; recheck each vendor's crawler page before you edit robots.txt.

In your control

• Per-bot robots.txt rules. Each agent is a separate line and a separate decision.
• Network reachability. OpenAI's help centre says eligibility needs both: allow OAI-SearchBot and "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." A firewall can return a 403 to a bot that robots.txt allows. From September 15, 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. A site can change status without anyone editing it.
• Rendering. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. Content that appears only after JavaScript runs may be invisible to them. Googlebot does render.
• Snippet controls. noindex, nosnippet, max-snippet:0 and data-nosnippet keep content out of AI answers on Google and Bing. Check they are not applied by accident.
• Plain facts on fetchable pages. A price, a number or a one-line definition gives an engine a sentence it can quote.

Not in your control

• What any shipped model already learned, and its cutoff.
• Whether an engine searches for a given question, and how it rewrites that question.
• Which retrieved pages the model chooses to cite.

That split is the whole strategy: fix the gates, and stop spending effort on what no one can move.

1. List the agents that matter to you. Most brands care about OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer and Googlebot for live answers.
2. Read your robots.txt line by line. Look for a blanket rule such as User-agent: * with Disallow: /, or a copied list of "AI bots" that includes search agents.
3. Decide training access separately. Allowing or blocking GPTBot, ClaudeBot, Google-Extended and Meta-ExternalAgent is a content-licensing choice. Make it deliberately and write down why.
4. Write explicit per-agent rules. A site that wants live AI visibility but declines training might use:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

5. Test the network gate. Request a page as each crawler and confirm you get a 200, not a 403 or a challenge page. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
6. Check rendering. Compare raw HTML with the rendered page. Move prices, product names and key claims into the server-rendered HTML.
7. Compare answers with search on and off. The free search on vs off test shows how far a training-only answer drifts from a search-backed one.

• Blocking every AI bot at once. A blanket block silently removes ChatGPT search eligibility along with training. This is the most common real-world error.
• Allowing GPTBot so ChatGPT can find you. GPTBot governs training only. ChatGPT search uses OAI-SearchBot.
• Assuming a GPTBot block hides you from ChatGPT. OpenAI states a webmaster can allow OAI-SearchBot while disallowing GPTBot.
• Trusting robots.txt alone. A CDN or firewall rule can override it without any visible sign.
• Expecting files or markup to get you into training. No engine documents reading llms.txt, and Google says structured data isn't required for its generative AI features.
• Expecting fast results from training access. Any effect waits for the next model, and nobody has measured its size.

For the live-answer side in more depth, see SEO for AI search.

Can I get my brand into an AI model's training data? You can allow training crawlers such as GPTBot and ClaudeBot, which signals that your content may be used. You cannot confirm inclusion: no vendor exposes training-corpus membership, and any effect appears only in a future model. No study has measured whether allowing training access improves brand recall. For the mechanics, see [how a brand ends up in training data](/questions/brand-in-ai-training-data).

Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended controls training and grounding for Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search. AI Overviews and AI Mode have no separate robots.txt token. Googlebot rules, indexing and snippet eligibility govern them, plus a Search Console control that Google is rolling out to a subset of site owners.

Will blocking GPTBot hurt my visibility in ChatGPT search? Not according to OpenAI. Its crawler page says each setting is independent, and gives the example of allowing OAI-SearchBot to appear in search results while disallowing GPTBot for training. What does remove you from ChatGPT search answers is opting out of OAI-SearchBot, or a firewall that blocks OpenAI's published search IP addresses.

Why does an AI answer about my company use outdated facts? The answer probably came from training data rather than a live search. Models answer from frozen knowledge when they choose not to search, and that knowledge stops at the model's cutoff. Different routes inside one product can carry different cutoffs. Making sure retrieval bots can reach your current pages, with facts in the server-rendered HTML, is the part you control.