If I allow AI search bots, will my content be used for training?

The major AI vendors split crawling into separate bots by purpose. Each bot reads its own line in robots.txt. So allowing a search bot does not grant a training bot permission, and blocking a training bot does not remove you from search.

Here is what each vendor documents, as of October 2026:

• OpenAI. GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot surfaces sites in ChatGPT search. OpenAI states: "Each setting is independent of the others." Its own example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot. Our GPTBot vs OAI-SearchBot guide covers the detail.
• Perplexity. PerplexityBot handles search indexing and, per Perplexity's crawler docs, is "not used to crawl content for AI foundation models."
• Anthropic. ClaudeBot collects training data. Claude-SearchBot supports search quality. Claude-User fetches pages when a user asks. See whether to allow Claude-SearchBot if you block ClaudeBot.
• Google. Google-Extended controls whether Gemini Apps and Vertex AI can use your content for training and grounding. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
• Meta. Meta-ExternalAgent is listed in our evidence base as a training crawler. Meta-WebIndexer powers Meta AI search.

The practical meaning: training and search are separate decisions, and you make them bot by bot.

This file allows the documented search crawlers and disallows the documented training crawlers. Adapt the paths if you only want to protect part of the site.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Google-Extended
Disallow: /

Three things to know before you ship it:

• Leave Googlebot alone. Google says robots.txt rules for Googlebot are the control for AI Overviews and AI Mode, because AI is built into Search. There is no separate token for those surfaces.
• The most common mistake is a blanket AI block. A single rule aimed at "AI bots" often catches OAI-SearchBot too, which removes ChatGPT search eligibility. OpenAI says opted-out sites can still appear as navigational links in ChatGPT, but not in search answers.
• Allowing a search bot buys eligibility. It does not guarantee retrieval, citation or traffic.

For a maintained version with comments, use the robots.txt template for AI search and training bots.

User-triggered fetchers

When a person asks an assistant to open your page, a different agent makes the request. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." Robots.txt governs crawling, not every fetch.

Google-Extended has a cost

Blocking Google-Extended cuts grounding for Gemini Apps as well as training. If you want Gemini Apps to draw on your live pages, weigh that before disallowing it. It does not touch Google Search.

Compliance is a stated policy, not a guarantee

Robots.txt is a request. Cloudflare notes that site owners may be unsure whether newer or less transparent crawlers respect it at all. The vendors above document compliance for their named crawlers. Unnamed scrapers are outside this pattern.

Past crawls stay past

A Disallow rule signals your preference from now on. It says nothing about content collected before you added it.

The vendor statements are first-party documentation, not independent measurement. They tell you which bot does what. They do not prove behaviour from outside.

Two gaps matter for the decision:

1. The effect of blocking training is unmeasured. OpenAI nowhere claims that allowing GPTBot improves search visibility. Nothing in the published evidence tests whether excluding training changes how well a model recalls your brand without web access. The honest label is unmeasured, not harmless and not harmful.
2. Training only reaches future models. Any training effect lands a model generation later, bounded by each model's knowledge cutoff. Search crawling affects answers that use live retrieval.

The one first-party statement linking crawler access to citation comes from Meta: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." No other vendor makes an equivalent promise.

So the decision is a values call about training use, plus a technical check that search bots can get in. The robots.txt file handles the first part. The second part needs testing.

Robots.txt is only the first gate. OpenAI says eligibility for ChatGPT search means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses. A firewall can return 403 to a bot your robots.txt allows, and nothing in a dashboard tells you.

1. Audit your robots.txt line by line. Confirm each search bot is allowed and each training bot matches your intent.
2. Check your CDN settings. From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and free-tier customers, while Search stays allowed. You may already have a training block you did not set.
3. Request your pages as each crawler. The free AI crawler checker shows which bots get through, and the firewall testing playbook walks through the fix.
4. For the full workflow, follow block AI training but stay in AI search.