Block AI Training but Stay in AI Search: Per-Vendor Steps

AI platforms use different crawlers for different jobs. A training crawler collects pages that may be used to build future models. A search crawler fetches pages so the engine can cite them in live answers.

These map to two different stages:

• Parametric knowledge is what a model knows with no web access. It is frozen at a training cutoff. Training crawlers feed this, a full model generation later.
• Access for retrieval decides whether a search crawler may fetch and store your page so it can enter answers today. Search crawlers feed this.

Blocking training touches the first stage only. Blocking search touches the second. If you want your content kept out of model training but still eligible for AI answers, you block one set of tokens and allow the other. Our guide to GPTBot vs OAI-SearchBot covers the OpenAI pair in detail.

Be clear about the trade. Nothing published measures how training exclusion affects what a model later says about you, in either direction. The honest label is unmeasured, not harmless and not harmful.

As of October 2026, per each vendor's crawler documentation:

Two vendor statements matter most. OpenAI says GPTBot is used for training and that disallowing it signals content should not be used for training, while OAI-SearchBot surfaces sites in ChatGPT search. Perplexity says PerplexityBot is for search indexing and is "not used to crawl content for AI foundation models," so there is no Perplexity training token to block.

Meta is the only vendor that ties access directly to citation: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

1. Open your live robots.txt at yourdomain.com/robots.txt. Edit the file your server actually serves, not a copy in a repo that never deployed.
2. Find any blanket AI block. A single group that disallows every AI bot is the most common error. It removes ChatGPT search eligibility along with training.
3. Give each bot its own group. Under the Robots Exclusion Protocol (RFC 9309), a crawler follows the group that names it and ignores the * group. Explicit groups remove ambiguity.
4. Add the rules. A minimal version:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

5. Check your CDN and firewall. OpenAI's help centre says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published searchbot IP addresses. Look for bot-management rules that challenge or 403 the search crawlers.
6. Check Cloudflare's defaults if you use it. From 2026-09-15, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stays allowed. Confirm the Search category is still allowed for your zone.
7. Record the date and the file. Save a copy of the robots.txt you shipped so you can diff it next time.

Google-Extended is the token people most often misread. Per Google's common crawlers documentation, blocking it cuts your content from Gemini Apps and Vertex AI for both training and grounding. Google also states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." The Google-Extended glossary entry has the full definition.

So blocking Google-Extended means:

• Gemini models may not train on your pages.
• The Gemini app and Vertex AI may not ground answers in your pages either. That is a live-answer cost, not only a training opt-out.
• Google Search, AI Overviews and AI Mode are unaffected.

There is no robots.txt token for AI Overviews or AI Mode. Googlebot is the control, and blocking it removes you from Google Search entirely. If you want out of AI Overviews specifically, see whether blocking Google-Extended removes you from AI Overviews and Google's AI features control in Search Console. That control is opt-out, on by default, and child properties inherit it from the parent, so check it was not switched off by accident.

Decide on Google-Extended with both effects in view. If Gemini app visibility matters to you, blocking it has a cost.

Run through this after every change:

• robots.txt returns HTTP 200 and is reachable by bots, not only by your browser.
• GPTBot, ClaudeBot, Google-Extended and Meta-ExternalAgent each resolve to Disallow.
• OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Googlebot and bingbot each resolve to Allow.
• No * group or leftover rule overrides a bot you meant to allow.
• A request to a real content page with each search crawler's user-agent returns 200, not 403, 429 or a challenge page.
• Cloudflare or other bot-management categories allow Search crawlers.
• The Search Console AI features control is still set to include, if you want AI Overviews.
• Key facts (pricing, product names) appear in the raw HTML, since most AI crawlers do not execute JavaScript.

The firewall line is the one most sites skip. Our answer on why robots.txt allows AI bots but you never show up in ChatGPT walks through it. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Good

• Every training token resolves to Disallow and every search token to Allow.
• Each search crawler gets a 200 on content pages and on robots.txt itself.
• Your server logs show verified search crawlers fetching pages after the change.

Good means eligible. It does not mean cited. Allowing a search crawler buys eligibility, not retrieval and not ranking.

Bad

• OAI-SearchBot blocked by a blanket rule. OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• robots.txt allows, firewall says 403. This is a silent, complete failure for that crawler.
• Googlebot blocked to stop AI. You lose all of Google Search, not just AI features.
• Google-Extended blocked without knowing it also cuts Gemini grounding.
• Expecting user-triggered fetchers to obey. ChatGPT-User, Perplexity-User and Meta-ExternalFetcher may not follow robots.txt, per their own vendors.

Crawler names and policies change, and your stack changes underneath you. Re-run the checklist:

• Immediately after any robots.txt edit, CMS migration, CDN switch or new bot-management rule.
• When a vendor updates its crawler page. New tokens appear, and the meaning of existing ones can shift.
• Quarterly as a baseline, even if nothing changed on your side. Cloudflare's 2026-09-15 defaults show access can change without you editing anything.

Keep expectations on training realistic. A training block applies to crawls from now on. Any effect on a model would show up a model generation later at the earliest, bounded by cutoffs: OpenAI lists Feb 16, 2026 for the GPT-5.6 family and Aug 31, 2025 for the Instant path. You will not see a measurable result from a training opt-out this quarter, and no published study shows one either way.

Will blocking GPTBot remove me from ChatGPT search? No. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the search crawler, and says each setting is independent. You can disallow GPTBot and allow OAI-SearchBot. Your site stays eligible for ChatGPT search as long as OAI-SearchBot can reach it through robots.txt and your firewall. Eligibility still is not a guarantee of being cited.

Does blocking AI training hurt how AI describes my brand? Nobody knows. No vendor and no published study has measured whether excluding your site from training changes what a model recalls about you. OpenAI does not claim allowing GPTBot improves visibility. Treat the training decision as a content-rights choice. Your live answers depend far more on search crawler access, which you can check today.

Can I stop user-triggered AI fetchers with robots.txt? Not reliably. OpenAI says robots.txt may not apply to ChatGPT-User because a person initiated the request. Perplexity says Perplexity-User generally ignores robots.txt. Meta documents that Meta-ExternalFetcher may bypass it. Blocking these at the firewall is possible, but it also stops pages being fetched when a user asks an assistant to read them.

Does Google penalize sites that block AI crawlers? Google states that Google-Extended is not used as a ranking signal and does not affect inclusion in Google Search. Blocking it is safe for Search. Blocking Googlebot is different: it removes you from Google Search and every AI surface built on it. More detail is in our answer on [whether Google penalizes blocking AI crawlers](/questions/google-penalize-blocking-ai-crawlers).

Is there a Perplexity training crawler to block? Perplexity's crawler documentation says PerplexityBot is used for search indexing and is not used to crawl content for AI foundation models. So the documented Perplexity token is a search token. Allow it if you want to stay eligible for Perplexity answers. Perplexity-User handles user-triggered fetches and generally ignores robots.txt.