Robots.txt for AI Crawlers: A Per-Bot Guide to Training and Search Bots

Most sites want two things at once: keep content out of model training, and stay eligible for AI search answers. Robots.txt can express that, because vendors split these jobs across different bots.

Here is a short snippet that opts out of training for OpenAI, Anthropic, Google's Gemini products and Meta, while allowing their search crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: *
Allow: /

One trade-off sits inside this file. Google documents that Google-Extended cuts both training and grounding for Gemini Apps and Vertex AI. Blocking it does not touch Google Search or AI Overviews, but it does remove you from Gemini Apps grounding. If Gemini app answers matter to you, drop that group.

Treat this as a starting point, not a universal answer. The robots.txt template for AI search and training bots covers more vendors and path-level variants.

Robots.txt is a plain text file at the root of each host. A crawler reads it, finds the group whose User-agent line matches its own name, and follows that group's rules. Under the Robots Exclusion Protocol (RFC 9309), a bot that matches a named group follows that group and ignores the * group. So naming a bot explicitly overrides your wildcard rules for that bot.

AI vendors run three kinds of bot, and the difference decides what a rule does:

• Training crawlers collect content that may be used to train foundation models. GPTBot, ClaudeBot and Meta-ExternalAgent sit here, and Google-Extended is the control token for Gemini.
• Search crawlers build the index an AI search product retrieves from at answer time. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer sit here.
• User-triggered fetchers visit a page because a person asked the assistant to. ChatGPT-User, Claude-User and Perplexity-User sit here.

OpenAI puts the principle in writing: "Each setting is independent of the others." A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. The digest of OpenAI's crawler documentation walks through each of its agents.

This table lists what each operator says happens when you disallow its bot, as of October 2026. Vendors rename and add agents, so recheck their pages before a major change.

The Anthropic split confuses people most. If you block training but want Claude's search to see you, read whether to allow Claude-SearchBot while blocking ClaudeBot. For Perplexity's agents, see the digest of Perplexity's crawler documentation.

The most common real-world error is a blanket AI block. Someone copies a list of "AI bots" into robots.txt and disallows all of them. That list usually includes OAI-SearchBot, PerplexityBot and Claude-SearchBot alongside the training crawlers.

The result is specific. OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." The same pattern removes you from Perplexity's index and from Claude's search crawling. You meant to stop training. You also left AI search.

The reverse mistake is just as common. Allowing GPTBot does not help you appear in ChatGPT search, because GPTBot governs training only. If your goal is search visibility without training use, block GPTBot and allow OAI-SearchBot. The question of whether to block GPTBot and still appear in ChatGPT search covers the edge cases.

A wildcard block (User-agent: * with Disallow: /) is worse still. Any bot without its own named group follows it, including Googlebot and Bingbot if you have not named them.

Robots.txt works at the access stage: whether a crawler may fetch your pages at all. That is a real lever, and it is in your hands. Several things sit outside it.

In your control

• Which named bots may fetch which paths on each host.
• Whether your firewall, CDN or bot manager lets those bots through. OpenAI's help centre says eligibility needs both: allow OAI-SearchBot and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses.
• Whether the robots.txt file itself is reachable. Anthropic warns that IP blocking can stop it reading your robots.txt at all.

Not in your control

• Retrieval and citation. Allowing a bot makes you eligible. It does not decide whether a page is retrieved for a given question, or cited.
• User-triggered fetchers. OpenAI and Perplexity both say their user agents may not follow robots.txt.
• What models already know. Training data is frozen at a cutoff. A robots.txt change today affects future collection, not past training.
• CDN defaults set by someone else. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and all existing free-tier customers. A site can lose AI access without anyone editing robots.txt.

If your file allows the right bots but you never show up, the checklist of what else can block ChatGPT is the next place to look.

1. Read your live robots.txt on every host. Each subdomain serves its own file, so www, blog and docs can disagree. See whether each subdomain needs its own rules.
2. Decide per vendor. For each of OpenAI, Anthropic, Google, Perplexity and Meta, decide training yes or no, and search yes or no. Write the decision down.
3. Write named groups. Give every bot you care about its own User-agent group. Do not rely on the wildcard to carry an AI decision.
4. Check your network layer. Look at your CDN or firewall bot settings for AI crawler categories. Make sure the search bots you allowed in robots.txt are not challenged or returned a 403.
5. Test as each bot. Request a page with each crawler's user agent and confirm a 200 response with real content. The playbook for testing robots.txt per AI bot shows how. The free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
6. Recheck after changes. Re-run the test after a CDN change, a migration or a new vendor bot announcement.

• Blocking every AI bot to stop training. This removes ChatGPT search eligibility. Block training bots by name instead.
• Allowing GPTBot for visibility. GPTBot governs training. OAI-SearchBot governs ChatGPT search.
• Expecting Google-Extended to remove you from AI Overviews. It does not. Google says AI is built into Search, so Googlebot rules are the control, and those also remove you from regular results. Google's separate Search Console control for generative features is opt-out by default and was still rolling out to a subset of owners as of its last documentation.
• Trusting robots.txt alone. A bot your file allows can still be blocked at the CDN. The two layers have to agree.
• Forgetting subdomains. A clean file on www says nothing about docs or shop.
• Adding files nobody reads. Google says Search ignores llms.txt, and no other engine documents parsing it. It is not a substitute for correct robots.txt rules.