robots.txt Template for AI Search and Training Bots

Each vendor runs separate crawlers, and each one reads its own group in robots.txt. The table below lists the bots named in the vendors' own documentation as of October 2026. Check the vendor pages before you publish, because bot names and roles change.

For the reasoning behind each label, see training crawlers vs search crawlers across vendors and the full per-bot robots.txt guide.

One syntax rule shapes every template below. A crawler follows the group that names it and ignores the User-agent: * group. So if your current file disallows private paths like /admin/ under *, copy those lines into every named group you add.

Use this if you want to be eligible everywhere and you accept training use. Listing the bots by name is optional when the * group already allows everything, but it documents intent and survives later edits.

# Policy: allow all AI bots (training, search, user-triggered)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

What it means: every documented search bot may fetch your pages. That buys eligibility, not retrieval or citation. Whether an engine uses your page for a given question is decided at answer time, and nobody outside the engine controls that.

This is the most common policy for brands that want to appear in AI answers but keep content out of model training. It works because the vendors separate the two jobs. OpenAI's crawler page says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, and that "each setting is independent of the others." The detail is in whether to block GPTBot and still appear in ChatGPT search.

# Policy: search only
# Training bots: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
Disallow: /

# Search bots: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
Allow: /

User-agent: *
Allow: /

Two trade-offs to decide

• Google-Extended covers more than training. Google says it also controls grounding for Gemini Apps and Vertex. Blocking it keeps you in Google Search, but Gemini app answers lose access to your pages. If Gemini app visibility matters more to you than training opt-out, delete Google-Extended from the Disallow group.
• User-triggered fetchers are left out on purpose. They fetch a page because a person asked for it. Blocking them is unreliable anyway, as the next template explains, and PerplexityBot vs Perplexity-User covers why.

Use this if you want no AI vendor crawling your site. It keeps Googlebot and bingbot allowed, because a blanket User-agent: * Disallow: / would also remove you from Google and Bing search.

# Policy: block all AI bots, keep classic search
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: *
Allow: /

What this file cannot do

• It cannot reliably stop user-triggered fetches. OpenAI says robots.txt "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." Meta documents that Meta-ExternalFetcher may bypass it. If you need a hard block, it has to happen at the firewall.
• It cannot block Google AI Overviews or AI Mode. No robots.txt token exists for them. Google treats AI as part of Search, so Googlebot rules are the control, and blocking Googlebot removes you from Search entirely. To keep content out of AI Overviews while staying indexed, Google documents nosnippet, max-snippet:0 and data-nosnippet, which also limit your normal search snippets. Google also offers a Search Console control for generative AI features. It is opt-out by default and, as of October 2026, still rolling out to a subset of site owners.

1. Pick one policy. Decide training, search and user-triggered access separately. If unsure about training, read whether allowing search bots means training use.
2. Fetch your live file. Open https://yourdomain.com/robots.txt and copy every existing Disallow line.
3. Merge, do not replace. Paste the template, then copy your existing private-path Disallow lines into each named group, since named bots skip the * group.
4. Publish at the root of every host. A robots.txt file applies to one host, so blog. and shop. subdomains need their own. See whether each subdomain needs separate rules.
5. Test how each bot resolves. Confirm that every named bot gets the Allow or Disallow you intended. The step-by-step test per AI bot walks through it, and the robots.txt Checker for AI Crawlers does it in one pass.
6. Test the network layer. Request a page with each search bot's user agent and record the status code. A 403 or a challenge page means your firewall or CDN blocks the bot whatever robots.txt says. OpenAI states eligibility needs both: allow OAI-SearchBot and confirm your host or CDN allows its published IP addresses. Cloudflare also changed its defaults on 15 September 2026, blocking Training and Agent crawlers on ad-displaying pages for new domains and free-tier customers.
7. Record the date and the policy. Write it in a comment at the top of the file so the next editor knows the intent.

• Each AI bot you care about is named in its own group or a shared group, with an explicit Allow or Disallow.
• Training bots and search bots are treated as separate decisions.
• OAI-SearchBot is not blocked by accident through a broad AI block, which is the most common real-world error.
• Googlebot and bingbot are still allowed, unless you intend to leave Google and Bing search.
• Existing private-path Disallow lines are copied into every named group.
• No User-agent: * Disallow: / line sits in the file by mistake.
• Every subdomain that serves content has its own robots.txt.
• Pages you want in AI answers do not carry noindex, nosnippet or max-snippet:0 inherited from a template.
• Each search bot receives a 200 from your firewall or CDN, not a 403 or a challenge.
• The file carries a dated comment stating the policy.

Good

• Every search bot you chose to allow resolves to Allow for your key pages, and a request with its user agent returns 200 with the real content.
• Every training bot you chose to block resolves to Disallow.
• Your server logs show allowed search bots fetching pages, and blocked training bots stopping after they read robots.txt.

Bad

• robots.txt allows OAI-SearchBot, but the CDN returns 403. This is the silent failure covered in why robots.txt allows AI bots but you never show up in ChatGPT.
• A search bot falls through to a * group that disallows everything.
• A named group allows / and forgets your /admin/ Disallow.
• robots.txt itself returns an error to some crawlers. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file."

How often to repeat

Re-run steps 5 and 6 after any change to robots.txt, your CDN, your firewall rules or your hosting plan. Re-check the vendor crawler pages every quarter, because vendors add and rename bots. A passing result means the gate is open. It does not guarantee any engine will cite you, which depends on choices made at answer time that you cannot see.

Does blocking GPTBot remove me from ChatGPT search? No. GPTBot governs training use only. ChatGPT search uses OAI-SearchBot, and OpenAI's crawler documentation says each setting is independent, so you can disallow GPTBot and allow OAI-SearchBot. The reverse also holds: allowing GPTBot does not make ChatGPT search more likely to find or cite you, and OpenAI claims no retrieval benefit from it.

Can I block Google AI Overviews with robots.txt? No token does that. Google treats AI Overviews and AI Mode as part of Search, so Googlebot rules are the control, and blocking Googlebot removes you from all of Google Search. Google-Extended does not affect AI Overviews. To limit what AI Overviews can show, use nosnippet, max-snippet:0 or data-nosnippet, or the Search Console generative AI control where it is available to you.

Will allowing every search bot get me cited in AI answers? No. An Allow makes your pages eligible to be fetched and indexed. Whether an engine retrieves your page for a specific question, and whether it cites it, is decided by the engine at answer time. What you control is removing blocks: robots.txt, firewall access, server-rendered content and snippet controls. Nobody outside the engine controls the rest.

Why not just use User-agent: * Disallow: / to block AI? Because the wildcard applies to every crawler that has no group of its own, including Googlebot and bingbot. That line removes you from Google and Bing search, and with them Google's AI features and the Bing index that backs Copilot. Name each AI bot explicitly instead, and leave the wildcard group allowing access.

Do I need to update the template when new AI bots appear? Yes. A bot with no named group follows your User-agent: * rules, which in these templates allow access. If you block AI by default, a new or renamed bot slips through until you add it. Check the OpenAI, Anthropic, Perplexity, Google and Meta crawler pages each quarter and add any new names to the right group.