Does blocking all AI bots in robots.txt hide me from AI search?

Most "block all AI bots" lists copied from blogs and plugins treat every AI user agent the same. They disallow GPTBot, ClaudeBot and Google-Extended, which are training crawlers. They also disallow OAI-SearchBot, PerplexityBot and Claude-SearchBot, which feed the engines' search features.

The consequence is documented. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Perplexity says PerplexityBot indexes for search and is not used to train foundation models. Block it and you leave Perplexity's index.

This failure is silent. No dashboard flags it, and the site keeps working for humans. If your goal was to keep content out of model training, a blanket block overshoots. It also removes you from the answer-time retrieval that most buyers actually see.

For the per-bot rules in full, see our per-bot guide to robots.txt for AI crawlers.

Each AI company runs several crawlers with separate jobs. As of October 2026, the operators document them like this:

OpenAI puts it plainly: "Each setting is independent of the others." A webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot. That also means allowing GPTBot does nothing for ChatGPT search, a point covered in whether to block GPTBot and still appear in ChatGPT search.

Meta is the only operator that ties access to citation in its own words: allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses."

Give each crawler its own group. Under the Robots Exclusion Protocol, a crawler follows the group that names it, so explicit groups are clearer than relying on a wildcard. A short, correct example:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

Think twice before adding Google-Extended. Google says it cuts training and grounding for Gemini Apps and Vertex, so blocking it removes you from grounded Gemini app answers. It does not affect Google Search inclusion and is not a ranking signal.

Also check that a stray User-agent: * group with Disallow: / isn't doing more than you meant. Our page on the Disallow directive and its common mistakes walks through the usual errors.

There is no robots.txt token that blocks Google AI Overviews or AI Mode. Google says AI is built into Search, so Googlebot directives are the control. A wildcard block that catches Googlebot removes you from Google Search and its AI features together.

Google states that a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. If you want to limit what appears in AI answers without leaving Search, the documented tools are on-page: nosnippet, max-snippet, data-nosnippet and noindex.

Search Console also has a generative AI inclusion control. As of the evidence date it is opt-out, on by default, and rolling out to only a subset of site owners.

robots.txt is necessary, not sufficient. Three gaps matter:

• User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." More in whether all AI crawlers obey robots.txt.
• Firewalls and CDNs. A WAF can return 403 to a bot your robots.txt allows. OpenAI's eligibility wording requires both the robots.txt allow and a host or CDN that lets its published IPs through. Since 2026-09-15, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and free-tier customers, while Search stays allowed. The robots.txt vs firewall comparison explains both gates.
• What models already know. Blocking search bots doesn't erase a model's training data. Answers given without live retrieval come from a frozen snapshot nobody can edit this quarter.

1. Open your robots.txt and list every AI user agent it names. Sort each into training or search.
2. Decide your training policy separately from your search policy. They are separate settings.
3. Give each search crawler you want an explicit Allow group.
4. Leave Googlebot alone unless you mean to leave Google Search.
5. Test whether each crawler can actually fetch a page. The robots.txt Checker for AI Crawlers shows what your file allows per bot.

Allowing search crawlers makes you eligible. It doesn't guarantee retrieval or a citation. For a step-by-step repair, follow the playbook for fixing a blanket AI block.