Training crawler: definition and what blocking it does

A training crawler is an automated bot run by an AI company to collect web content that may be used to train its generative models. It feeds the model's built-in knowledge, not the live web results an assistant fetches when it searches.

OpenAI describes GPTBot this way: it crawls "content that may be used in training our generative AI foundation models," and disallowing it "indicates a site's content should not be used in training." Anthropic runs ClaudeBot for the same purpose. Meta runs Meta-ExternalAgent. Google uses a robots.txt token, Google-Extended, to let sites opt out of training and grounding for Gemini Apps and Vertex AI.

For a vendor-by-vendor breakdown, see training crawlers vs search crawlers.

Most AI vendors split their bots by job. One collects training data. Another builds a search index. A third fetches pages when a user asks. You control each one with its own robots.txt rule.

Perplexity states PerplexityBot is for search indexing and is not used to crawl content for foundation models.

A site that wants to stay out of OpenAI training but stay in ChatGPT search would write:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

robots.txt is only one gate. A CDN or firewall can also turn crawlers away. Cloudflare began blocking Training and Agent crawlers by default on ad-displaying pages from 2026-09-15 for new domains, new sites on existing accounts and existing free-tier customers. That change could alter your training policy without anyone on your team editing a file.

Training crawlers only touch the parametric stage: what a model knows with no web access. Any effect arrives a full model generation later, and it is capped by each model's knowledge cutoff. As of October 2026, OpenAI documents a Feb 16, 2026 cutoff for the GPT-5.6 family and Aug 31, 2025 for the Instant path, so "what ChatGPT knows" is not one value.

Nothing published tests whether blocking training degrades how a model recalls a brand, or whether allowing it helps. The honest label is unmeasured, in both directions. We cover the question in depth in whether blocking training bots makes ChatGPT forget your brand.

What you can control is the separation. Decide your training policy on its own terms (licensing, legal, preference). Then confirm your search crawlers are still allowed, because those govern whether you can appear in live AI answers. A hand-built robots.txt for AI bots makes each rule explicit.

• "Allow GPTBot so ChatGPT can find you." GPTBot governs training only. OpenAI asserts no search benefit from it. ChatGPT search uses OAI-SearchBot.
• "Blocking GPTBot removes you from ChatGPT." OpenAI says "each setting is independent of the others." You can block training and still appear in search answers.
• "Blocking Google-Extended removes you from AI Overviews." Google says Google-Extended does not affect inclusion in Google Search. There is no robots.txt token for AI Overviews or AI Mode. See Google-Extended and AI Overviews.
• "Blocking all AI bots is the safe default." A blanket block silently removes search eligibility too. Read what blocking all AI bots in robots.txt does before you ship one.

• Search crawler: a bot that builds an index used for live AI answers, such as OAI-SearchBot or Claude-SearchBot.
• User-triggered fetcher: a bot that fetches a page when a user asks, such as ChatGPT-User. OpenAI says robots.txt rules may not apply to it.
• Bot management: CDN and firewall rules that can block crawlers regardless of robots.txt.
• Network blocking: when a firewall returns a 403 to a crawler robots.txt allows. See when your CDN or firewall blocks AI crawlers.

Which bots count as AI training crawlers? The main ones are OpenAI's GPTBot, Anthropic's ClaudeBot and Meta's Meta-ExternalAgent. Google handles training through the Google-Extended robots.txt token rather than a separately named fetcher. Each vendor documents its bots on its own page, and the list changes, so check the vendor pages before writing rules rather than copying an old block list.

Does blocking training crawlers stop AI from using my content entirely? No. It tells that vendor not to train future models on your pages. It does not remove content already collected, and it does not stop search crawlers or user-triggered fetchers. OpenAI notes robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules.

Should I block or allow training crawlers? The evidence cannot settle that for visibility, because the effect on brand recall has never been measured. Treat it as a business decision about how your content is used. Whatever you choose, keep the decision separate from search access, and confirm your search crawlers still get through both robots.txt and your firewall.

Could my site be blocking training crawlers without my knowledge? Yes. A CDN default, a security plugin or a bot-management rule can block crawlers even when robots.txt allows them. Cloudflare's 2026-09-15 default change is one example of a third party altering access with no edit on your side. The fix is to request your pages as each crawler and compare the response with your robots.txt.