Do all AI crawlers obey robots.txt?

AI companies run two different kinds of bots, and they treat robots.txt differently.

• Index-time crawlers visit pages ahead of time to build a search index or a training set. OpenAI, Perplexity and Anthropic document these bots and describe robots.txt as the way to control them.
• User-triggered fetchers go to a page because a person asked the assistant to open it right now. OpenAI and Perplexity both say robots.txt may not govern these.

The reasoning the operators give is that a user-triggered fetch acts on behalf of a person, more like a browser than a crawler. As of October 2026, OpenAI's crawler page says that for ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity's documentation says Perplexity-User "generally ignores robots.txt rules."

Robots.txt itself is a convention, standardized as the Robots Exclusion Protocol. A crawler follows it because its operator chose to. Nothing in the file stops a request. So the practical answer is: robots.txt controls the documented crawlers that say they honor it, and anything stricter needs a block at the network layer.

Here is what each operator documents, as of October 2026. Treat it as the operator's statement, not an independent audit.

Two details matter. First, each rule is per bot. OpenAI says "Each setting is independent of the others," so blocking GPTBot does not remove you from ChatGPT search. Our per-bot guide to robots.txt for AI crawlers covers each token. Second, Google has no separate robots.txt token for AI Overviews or AI Mode. Googlebot rules are the control there, and Google-Extended covers Gemini training and grounding.

For platform detail, see how OpenAI's crawlers interact with robots.txt and firewalls and PerplexityBot vs Perplexity-User.

An operator saying a bot obeys robots.txt is a claim. You can't see from your robots.txt whether the claim holds. Only your server logs show what actually arrived.

Cloudflare, which sees traffic across a large share of the web, describes the question as open. Its Content Signals announcement notes that site owners "might be unsure whether these bots even respect robots.txt rules, especially newer or less transparent crawlers." No study in our evidence base measures how often AI crawlers break robots.txt rules, so we give no rate.

There is also a coverage gap. A crawler that is not named in your file falls under your wildcard rule, if you have one. If you don't, it is allowed by default. See what happens when robots.txt doesn't mention a bot.

Use a firewall, CDN or bot-management rule when robots.txt can't do the job:

• You want to stop user-triggered fetchers such as ChatGPT-User or Perplexity-User.
• You want to stop an undocumented or unverified crawler.
• You need a block that holds even if a bot ignores the file.

The firewall cuts both ways. OpenAI's help center says eligibility for ChatGPT search means allowing OAI-SearchBot in robots.txt and confirming your host or CDN allows traffic from OpenAI's published searchbot IP addresses. A bot-protection system can return a 403 to a crawler your robots.txt welcomes, and nothing in any dashboard warns you. Anthropic adds that IP blocking impedes its ability to read your robots.txt at all.

Defaults can change without you editing anything. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stayed allowed by default.

Blocking user fetchers has a cost: when someone asks an assistant to read your page, it can't. Decide per bot. Also note that blanket blocks do more than people expect, as covered in whether blocking all AI bots hides you from AI search.

1. List the bots you care about by purpose. Separate training (GPTBot, ClaudeBot, Google-Extended), search indexing (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user fetchers (ChatGPT-User, Perplexity-User).
2. Write explicit per-bot rules. Don't rely on a single wildcard to express different intentions.
3. Confirm crawlers can read the file. A blocked /robots.txt means bots can't learn your rules. The free robots.txt reachability check covers this.
4. Test what your server returns to each bot. Request a page with each crawler's user agent and look for 403s, challenges or CAPTCHAs. The AI Crawler Checker does this, and our playbook for testing rules per AI bot walks through it by hand.
5. Enforce hard blocks at the edge. Validate on user agent plus your CDN's verified-bot programme rather than IP lists alone, which OpenAI warns against relying on.
6. Recheck after any CDN or plan change. Defaults move.

Does blocking ChatGPT-User in robots.txt stop ChatGPT from opening my page? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person triggered the request. It also says ChatGPT-User is not used to decide whether content appears in ChatGPT search. If you need to stop these fetches, block the user agent at your firewall or CDN, and accept that users asking ChatGPT to read your page will then get nothing.

Does PerplexityBot respect robots.txt? Perplexity documents PerplexityBot as its search indexing crawler, controllable through robots.txt, and says it is not used to train foundation models. Perplexity-User is the exception: Perplexity says it generally ignores robots.txt because it fetches pages during a user's query. For the full breakdown, see [does Perplexity respect robots.txt](/questions/does-perplexity-respect-robots-txt).

If I block GPTBot, will I disappear from ChatGPT search? No. GPTBot governs training use only. OAI-SearchBot controls whether your site is shown in ChatGPT search answers, and OpenAI states each setting is independent. You can disallow GPTBot and still allow OAI-SearchBot. The reverse also holds: allowing GPTBot does nothing for search visibility.

Can my firewall block an AI crawler that robots.txt allows? Yes, and it happens silently. Bot-management and WAF systems can return a 403 Forbidden or a JavaScript challenge to a crawler your robots.txt permits. OpenAI says ChatGPT search eligibility requires both robots.txt permission and a host or CDN that allows its published searchbot IPs. The only way to know is to request your pages as each crawler and read the response.

Is there a robots.txt rule that removes my site from Google AI Overviews? There is no separate token for AI Overviews or AI Mode. Google says Googlebot rules are the control, because AI is built into Search. Page-level controls such as noindex, nosnippet, max-snippet and data-nosnippet limit what appears in AI features. Google-Extended affects Gemini training and grounding, not Search inclusion.