Free robots.txt Rules for AI Bots, Generated by Hand

A robots.txt check answers one narrow question: for each named AI crawler, does your file say yes or no? That matters because AI vendors run several bots with different jobs. One trains models. Another builds a search index. A third fetches pages when a user asks.

OpenAI's crawler documentation is explicit: GPTBot crawls content that may be used for training, OAI-SearchBot surfaces sites in ChatGPT search, and "each setting is independent of the others." Anthropic splits ClaudeBot (training) from Claude-SearchBot (search quality) and Claude-User (user-triggered). Perplexity says PerplexityBot is for search indexing and is not used to crawl content for AI foundation models.

So a single blanket "block AI" rule usually does more than its author intended. The most common real error is a training block that also removes search eligibility. Our training crawlers vs search crawlers comparison lays out the split vendor by vendor.

Pick your policy first, then write one group per bot. These steps produce the common policy: block training, allow search.

1. Open your current file at https://yourdomain.com/robots.txt and copy it. Keep any existing rules for Googlebot, Bingbot and your sitemap.
2. Decide per purpose, not per vendor. Training, search indexing and user-triggered fetching are separate choices.
3. Add a group for each crawler you want to name. A crawler follows the most specific group that matches its name, so a named group overrides the * group for that bot (RFC 9309 defines this behavior).
4. Upload the file to the root of each host. www and the bare domain need their own file if they serve separately.
5. Test each bot against the live file before you call it done.

Here is a short, correct snippet for the block-training, allow-search policy:

# Training crawlers: block
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# AI search crawlers: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Two choices deserve a second look. Google says Google-Extended cuts training and grounding for Gemini Apps and Vertex, and does not affect inclusion in Google Search. Meta states that "allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." For partial access, see how Allow and Disallow rules interact for AI crawlers. A fuller version lives in our robots.txt template for AI search and training bots.

There is no robots.txt token that blocks or allows Google AI Overviews or AI Mode. Google treats AI as part of Search, so Googlebot rules are the control.

Google's AI features documentation says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link. That gives you two practical levers:

• Leave Googlebot allowed if you want to stay eligible.
• Use noindex, nosnippet, max-snippet or data-nosnippet to keep specific pages or passages out of AI answers.

Blocking Google-Extended does not remove you from AI Overviews. Blocking Googlebot removes you from Google Search entirely, which is rarely what anyone wants.

Testing has two layers: what the file says, and what your server actually returns to each bot.

For the file, read it as each crawler would. Find the group that matches the bot's name, then check the rule for the path. Our step-by-step guide to testing robots.txt rules for each AI bot walks through it.

For the server, request a page with each crawler's user-agent string, copied from the vendor's documentation:

curl -s -o /dev/null -w "%{http_code}" -A "<full OAI-SearchBot user-agent>" https://yourdomain.com/

Read the status code like this:

• 200: the request got through.
• 403: a firewall or bot-management rule is turning the bot away, whatever robots.txt says.
• 429: rate limiting.
• A challenge or CAPTCHA page: a fail, because crawlers cannot solve it.

OpenAI's help centre says eligibility needs both halves: allow OAI-SearchBot and confirm your host or CDN allows traffic from its published IP addresses, listed at openai.com/searchbot.json. A test from your own laptop shows how your firewall treats the user-agent, not the bot's real IPs. Your server logs are the ground truth.

robots.txt is a request, and it controls less than many guides imply. Be clear about what sits outside your file.

• User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules."
• Firewalls and CDNs. As of October 2026, Cloudflare blocks training and agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers, a change that took effect on 15 September 2026. Your site can lose access without anyone editing robots.txt.
• Eligibility is not inclusion. Allowing OAI-SearchBot makes you eligible for ChatGPT search. It does not get you retrieved, cited or ranked.
• Training effects are unmeasured. Nothing published shows that allowing GPTBot helps or hurts how ChatGPT describes you.

If the file is right and you still never appear, work through what else can block ChatGPT when robots.txt allows it.

Hand-checking eight or more bots across two hosts is slow and easy to get wrong. The robots.txt Checker for AI Crawlers from 99Visibility parses your live file and shows the verdict per bot, so you can confirm the policy you wrote is the policy you shipped.

Run it after every deploy, and again after any CDN or firewall change. Pair it with a live request test, because the file and the firewall can disagree. For the reasoning behind each bot choice, read our per-bot guide to robots.txt for AI crawlers.

Will blocking GPTBot remove me from ChatGPT search? No. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the crawler for ChatGPT search, and says each setting is independent. You can disallow GPTBot to opt out of training and still allow OAI-SearchBot to stay eligible for search answers. Sites that block OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

Should I allow Claude-SearchBot if I block ClaudeBot? If you want to stay eligible for Claude's search features, yes. Anthropic separates ClaudeBot, which collects training data, from Claude-SearchBot, which it uses for search quality, and Claude-User, which fetches pages for user requests. Each reads its own robots.txt group. Blocking one does not block the others unless you write a rule for each or use a blanket wildcard block.

Does a User-agent: * Disallow rule block all AI bots? Only bots that have no group of their own. Under RFC 9309, a crawler follows the most specific group matching its name. If you add a named group for OAI-SearchBot with Allow, that bot ignores the wildcard. A wildcard Disallow also blocks Googlebot and Bingbot, which removes you from regular search, so it is rarely the right tool for an AI policy.

Can I allow an AI crawler on some pages but not others? Yes. Inside a bot's group, add Disallow lines for the paths you want closed and Allow lines for the paths you want open. The longest matching path wins when rules conflict. For example, you could disallow /members/ for PerplexityBot while allowing everything else. Test a URL from each path after deploying, because one typo in a path silently changes the result.

Is robots.txt enough to keep content out of AI answers? Not always. User-triggered fetchers such as Perplexity-User and ChatGPT-User may not follow robots.txt, by their vendors' own statements. For Google's AI features, robots.txt has no AI-specific token, so you use noindex, nosnippet, max-snippet or data-nosnippet on the page. Bing also documents data-nosnippet as excluding content from snippets and AI summaries while keeping it indexed.