Fixing a Blanket AI Block in robots.txt Without Losing AI Search

The most common robots.txt error we see described in AI visibility work is simple. Someone wanted to stop AI companies training on their content, so they disallowed every AI user agent they could find. That list usually includes the search bots too.

The result is silent. Nothing breaks on your site. Your Google traffic looks normal. But ChatGPT search, Perplexity and Claude search lose the right to index your pages.

OpenAI's crawler documentation is explicit about the consequence: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Perplexity says PerplexityBot handles search indexing and is "not used to crawl content for AI foundation models." Anthropic separates ClaudeBot (training) from Claude-SearchBot (search quality).

So a block meant to stop training also removes search eligibility. If you want the background on whether a full block hides you, see does blocking all AI bots hide me from AI search.

The blanket block usually takes one of three forms:

• A long list of AI user agents, each with Disallow: /.
• A User-agent: * group with Disallow: /, which blocks Googlebot and everything else too.
• A plugin or CDN toggle labelled something like "block AI bots" that writes rules or firewall policies you never read. WordPress security plugins are a frequent source.

The fix depends on knowing which bot does what. As of October 2026, the vendors document these roles:

OpenAI's bots page puts the key point in one line: "Each setting is independent of the others." It gives the exact example of allowing OAI-SearchBot to appear in search while disallowing GPTBot for training. Our page on whether to block GPTBot and still appear in ChatGPT search covers that case in detail, and the wider comparison of training and search crawlers lists more vendors.

Two notes before you edit anything:

• Google has no robots.txt token for AI Overviews or AI Mode. Blocking Googlebot removes you from Google Search entirely. Google-Extended does not affect Search inclusion. See the controls Google does offer for AI Mode.
• User-triggered fetchers behave differently. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Don't rely on robots.txt to control them.

1. Fetch your live file. Open yourdomain.com/robots.txt in a browser. Edit the file the server actually returns, not a copy in your CMS that may be overridden.
2. Find the blanket rule. Look for AI user agents grouped with Disallow: /, and for a User-agent: * group that disallows the root. Our guide to the Disallow directive explains how path matching works.
3. Decide your policy per vendor. Write down, for each vendor, whether you want training use and whether you want search eligibility. These are two separate decisions.
4. Give each bot its own group. Under RFC 9309, a crawler follows the group that names it most specifically. A bot named in its own group ignores the * group, so explicit groups remove ambiguity.
5. Write the split file. A typical training opt-out with search kept open looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: *
Allow: /

6. Keep genuine private paths blocked. If you disallow /admin/ or /checkout/ for everyone, repeat those lines inside each named group. A named group does not inherit rules from *.
7. Deploy and re-fetch. Load the live file again and confirm the new version is served, including through any CDN cache.
8. Check the second gate. OpenAI's help centre says eligibility needs both: allow OAI-SearchBot and confirm your host or CDN allows traffic from OpenAI's published search bot IP addresses. Request a page as each search bot and confirm a 200 response.

Run through this list after every change:

• The live robots.txt returns HTTP 200, not a 403, 404 or challenge page.
• OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer each have an explicit group with Allow: / (or no disallow on your public pages).
• GPTBot, ClaudeBot and Google-Extended reflect your actual training decision.
• Googlebot is not blocked from public pages.
• Private paths are repeated inside every named group that needs them.
• No plugin, theme or CMS setting regenerates the old file on the next update.
• Your firewall or CDN returns 200 to each search bot, not 403, 429 or a CAPTCHA.
• Your CDN bot-management categories treat Search crawlers as allowed.

The firewall items matter as much as the file. The comparison of robots.txt and firewall rules explains why the two gates often disagree. Anthropic also warns that IP blocking can stop it reading your robots.txt at all, which is covered in whether a blocked crawler can still read robots.txt. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Good

• Each search bot gets a 200 response on both robots.txt and a sample content page.
• Your robots.txt states a deliberate, per-vendor policy you can explain in one sentence each.
• Training bots are blocked or allowed by choice, not by accident.

Bad

• robots.txt allows OAI-SearchBot, but the CDN returns 403. This is a complete, silent failure for ChatGPT search.
• A User-agent: * group disallows the root and no search bot is named. Every compliant bot, including Googlebot, is shut out.
• The search bots are allowed, but your key facts only appear after JavaScript runs. Most AI crawlers other than Googlebot don't execute JavaScript, so they fetch an empty shell.

What the fix does not do

Allowing a search bot makes you eligible to be indexed. It does not make the engine retrieve you for a given question, and it does not guarantee a citation. Retrieval is decided per query by each engine. Treat this playbook as removing a block, then measure each engine separately to see what changes.

robots.txt drifts more than people expect. Run the checklist on this schedule:

• After any change to your CMS, security plugin, CDN plan or hosting provider.
• Monthly as a routine check, because vendors add new bot names and plugins update their default lists.
• Immediately after a known third-party default change. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search stays allowed by default, but confirm your own zone settings.

A quick way to spot regressions between full checks is the free ChatGPT search block check in this library. For a broader snapshot, the free checker on this site shows whether AI engines name you for up to three buyer questions once access is fixed.

Will blocking GPTBot remove me from ChatGPT search? No. GPTBot governs training use only. ChatGPT search uses OAI-SearchBot, and OpenAI states that each setting is independent of the others. You can disallow GPTBot to signal that your content should not be used for training, and allow OAI-SearchBot so your pages remain eligible to appear in ChatGPT search answers. Blocking OAI-SearchBot is what removes you from search results.

Does allowing GPTBot help me get cited in ChatGPT? There is no evidence that it does. OpenAI's documentation describes GPTBot as a training crawler and claims no search benefit from allowing it. Any effect on what the model knows would arrive with a future model generation and has never been measured. If your goal is ChatGPT search eligibility, the bot that matters is OAI-SearchBot, plus a firewall that lets it through.

Can I block Google AI Overviews in robots.txt without leaving Google Search? No. Google has no robots.txt token for AI Overviews or AI Mode. Googlebot controls all Google Search surfaces, so blocking it removes you from Search entirely. Google-Extended controls Gemini Apps training and grounding and does not affect Search inclusion. For limiting what appears in AI features, Google documents page-level controls such as nosnippet, max-snippet and data-nosnippet.

Does robots.txt stop Perplexity and ChatGPT from fetching pages a user asks about? Not reliably. Perplexity says Perplexity-User generally ignores robots.txt rules, and OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiates those fetches. robots.txt controls index-time crawling by bots such as PerplexityBot and OAI-SearchBot. Answer-time fetching by user agents is a separate behaviour that robots.txt does not fully govern.

I fixed robots.txt. Why can't AI search bots still reach my site? The most likely cause is the network layer. A firewall, CDN or bot-management rule can return 403, 429 or a challenge page to a crawler that robots.txt allows. OpenAI says eligibility requires both allowing OAI-SearchBot and letting its published IP addresses through your host or CDN. Request a page as each search bot and check the status code.