Training Crawlers vs Search Crawlers Across AI Vendors
99Visibility Research · Updated October 21, 2026 · 6 min read
Short answer
A training crawler collects pages that may be used to train a vendor's models. A search crawler indexes pages so the vendor's AI search can retrieve and cite them. OpenAI, Anthropic and Meta run a separate bot for each job, and each robots.txt rule is independent. Blocking training bots does not remove you from AI search, but a blanket block on every AI bot does. No one has measured a visibility benefit from allowing training crawlers.
Key takeaways
- Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) and search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer) are controlled by separate robots.txt rules.
- You can block AI training and stay eligible for AI search, because OpenAI states each crawler setting is independent of the others.
- A blanket block on every AI user agent also blocks the search crawlers, which removes your eligibility for those engines' search answers.
- Google has no robots.txt token for AI Overviews or AI Mode: Googlebot is the control there, and Google-Extended covers Gemini training and grounding only.
- Nobody has measured whether allowing training crawlers improves how AI describes your brand, so treat that choice as a policy decision, not a visibility lever.
What separates a training crawler from a search crawler
The two kinds of bot feed different parts of an AI system.
- Training crawlers collect pages that may go into a future model's training data. Anything they gather only shows up in what the model "knows" with no web access, and only after a new model is trained past its knowledge cutoff.
- Search crawlers build the index an AI search feature retrieves from at answer time. If a search crawler can't fetch your page, that engine can't put it in front of the model when someone asks a question.
There's a third group: user-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User. They fetch a page when a person asks the assistant to, and they follow different rules (covered below).
What this means: your robots.txt decision is really two decisions. One is about training use. The other is about whether you can appear in AI search answers at all. Make them separately.
The vendor map
As of October 2026, this is how each vendor splits its crawlers, based on each vendor's own crawler documentation.
| Vendor | Training crawler | Search or index crawler | User-triggered fetcher |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot (ChatGPT search) | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot (Claude search) | Claude-User |
| Perplexity | None named | PerplexityBot (Perplexity index) | Perplexity-User |
| Meta | Meta-ExternalAgent | Meta-WebIndexer (Meta AI search) | Meta-ExternalFetcher |
| Google-Extended (a robots.txt token, not a separate bot) | Googlebot (all Google Search surfaces) | None separate |
Three details matter.
- Perplexity states PerplexityBot is "not used to crawl content for AI foundation models." It is a search crawler only.
- Google-Extended isn't purely a training switch. Blocking it cuts training and grounding for Gemini Apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." The Gemini crawler controls page covers this in detail.
- Meta's documentation contains the only first-party statement tying a crawler to citation: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." For Anthropic's three agents, see ClaudeBot, Claude-SearchBot and Claude-User explained.
Each setting is independent
OpenAI states it directly: "Each setting is independent of the others." Its example is a webmaster who allows OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. The digest of OpenAI's crawler documentation walks through each agent.
The same logic applies across vendors because each bot reads its own User-agent group in robots.txt. So:
- Blocking GPTBot doesn't remove you from ChatGPT search.
- Allowing GPTBot doesn't put you in ChatGPT search.
- Blocking ClaudeBot says nothing about Claude-SearchBot.
- Blocking Meta-ExternalAgent leaves Meta-WebIndexer untouched. See blocking Meta's training crawler while staying in Meta AI.
The search consequence is also specific. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Allowing a search crawler buys eligibility. It doesn't guarantee retrieval, a citation or a position.
How a blanket AI block removes search eligibility
The error to look for is a robots.txt written to "block AI" that lists every AI user agent, or a wildcard rule that catches them. It stops training use, but it also disallows OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer. Those engines then can't index your pages for their search answers.
robots.txt isn't the only gate. OpenAI's help centre says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from OpenAI's published searchbot IP addresses. A firewall can return 403 to a crawler that robots.txt allows, and nothing in your dashboards will show it.
Defaults can also change under you. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, with Search left allowed. A site can lose access it never chose to lose.
The next move: read your robots.txt group by group, then test each crawler at the network layer. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
What allowing training crawlers does and does not do
Here the honest answer is short: nobody knows the effect.
OpenAI describes GPTBot as crawling content "that may be used in training" and claims no retrieval or visibility benefit from allowing it. No study in our evidence base tests whether excluding a site from training changes how a model later recalls a brand. The correct label is unmeasured, not helpful and not harmful.
Timing limits it further. Training data only reaches answers given without live retrieval, and only after the next model ships. OpenAI lists a Feb 16, 2026 knowledge cutoff for its GPT-5.6 models and Aug 31, 2025 for the Instant path. A page published after a model's cutoff can't be in that model's built-in knowledge, whatever your robots.txt says.
Blocking now also doesn't reach backwards. Pages already collected may sit in existing models. See what happens to pages an AI crawler already fetched.
What this means: decide on training crawlers based on how you want your content used. Don't decide based on a promised visibility gain, because none has been shown.
Which to use when
You want AI search visibility and don't mind training use
Allow all search crawlers and all training crawlers. Then confirm your CDN doesn't block them.
You want AI search visibility but no training use
This is the most common goal, and the rules support it. Disallow the training agents and explicitly allow the search agents:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Meta-WebIndexer
Allow: /
Add Google-Extended to the disallow list only if you also accept losing Gemini Apps grounding. The block AI training but stay in AI search playbook and the robots.txt template for AI bots cover the full set.
You want out of Google AI Overviews and AI Mode
robots.txt can't do this without also blocking Googlebot from Search. Use the snippet controls (nosnippet, max-snippet, data-nosnippet) or the Search Console generative AI control instead.
You want to block user-triggered fetching
Don't rely on robots.txt. OpenAI says it "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Only the network layer can stop these fetchers.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot governs training use only. ChatGPT search is controlled by OAI-SearchBot, and OpenAI states each crawler setting is independent of the others. You can disallow GPTBot and allow OAI-SearchBot in the same robots.txt. If you block OAI-SearchBot, OpenAI says your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link.
Can I block Google AI Overviews with a robots.txt rule?
Not separately from Search. Google has no robots.txt token for AI Overviews or AI Mode, and Google-Extended doesn't affect them. Googlebot governs all Google Search surfaces, so blocking it removes you from Search entirely. To limit what appears in AI answers, use nosnippet, max-snippet, data-nosnippet or noindex, or the generative AI control in Search Console where it's available to your property.
Do user-triggered fetchers like ChatGPT-User follow robots.txt?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiates the request. Perplexity says Perplexity-User generally ignores robots.txt rules, and Meta documents that Meta-ExternalFetcher may bypass them. If you need to stop these fetchers, the control is your firewall or CDN, not robots.txt. Blocking them doesn't affect training or search indexing.
Will allowing training crawlers make AI mention my brand more?
There's no evidence either way. No vendor claims that allowing a training crawler improves visibility, and no published study has measured the effect of training inclusion on how models describe a brand. Any effect would only appear after a future model is trained, and only in answers given without live search. Treat it as a content-use decision rather than a visibility tactic.
Is Google-Extended the same as blocking Gemini training only?
No. Google says blocking Google-Extended cuts training and grounding for Gemini Apps and Vertex AI. So it can affect whether Gemini Apps use your pages in live answers, not just future training. Google also says it doesn't affect inclusion in Google Search and isn't a ranking signal, so AI Overviews and AI Mode are unaffected by it.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.