Disallow directive: definition and the blanket AI block mistake

Disallow is one of the core rules in the Robots Exclusion Protocol, standardised as RFC 9309. It sits inside a group that starts with a User-agent line. The crawler named in that line should not request any URL whose path begins with the value after Disallow:.

• Disallow: / blocks the whole site for that bot.
• Disallow: /private/ blocks one folder.
• An empty Disallow: blocks nothing.

This entry is a short definition. For every AI bot, its token and what blocking it costs you, read the per-bot robots.txt guide for AI crawlers.

A crawler reads /robots.txt, finds the group that names it, and follows only that group. A User-agent: * group applies to any crawler that has no group of its own. When Allow and Disallow rules both match a path, the more specific rule wins, which is covered in how Allow and Disallow interact for AI crawlers.

Here's a correct snippet for a site that wants to stay out of model training but stay eligible for AI search:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

As of October 2026, OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search. Google documents Google-Extended as the control for Gemini training and grounding, and says it does not affect inclusion in Google Search.

Disallow is one of the few levers with a documented consequence, and it works by removing access. OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." That is a direct, first-party effect of one line in your file.

The reverse does not hold. Allowing a bot buys eligibility, not citations or rankings. What the engine retrieves and cites for a given question is outside your control.

The common error is the blanket AI block. A team wants to stop training use, copies a list of "AI bots" into robots.txt, and disallows all of them. That list often includes OAI-SearchBot, PerplexityBot and Claude-SearchBot, so the site drops out of AI search while the team believes it only opted out of training. Some teams go further with User-agent: * and Disallow: /, which also blocks Googlebot and every other crawler without its own group.

The next move is to read your live file group by group. The playbook for fixing a blanket AI block walks through the rewrite. The robots.txt Checker for AI Crawlers shows how each AI bot resolves against your current rules.

• "Blocking GPTBot hides me from ChatGPT." It doesn't. OpenAI says each setting is independent of the others. See whether to block GPTBot if you want ChatGPT search.
• "Disallow hides a page from AI answers." Disallow controls fetching. The documented controls for keeping content out of Google and Bing AI answers are noindex, nosnippet, max-snippet and data-nosnippet, explained in the robots meta tag entry.
• "There's a Disallow token for AI Overviews." There isn't. As of October 2026, Googlebot rules govern all Google Search surfaces, as covered in AI Overviews and Googlebot.
• "Every AI fetcher obeys Disallow." OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules."
• "Allow means the bot gets in." OpenAI's help centre says eligibility also requires your host or CDN to allow its published searchbot IP addresses. If you allow bots and still never appear, check what else could block ChatGPT.

• User-agent: the line that names which crawler a group applies to.
• Allow: the rule that permits paths inside a disallowed folder.
• Robots meta tag: a page-level control for indexing and snippet display, separate from robots.txt.
• Training bot vs search bot: GPTBot, ClaudeBot and Google-Extended govern training use. OAI-SearchBot, PerplexityBot and Claude-SearchBot govern search eligibility.
• Firewall or CDN block: a network-level refusal that can stop a crawler your robots.txt allows.

Does Disallow remove a page that is already indexed? Not on its own. Disallow tells a crawler not to fetch the URL, which is a crawling instruction. OpenAI notes that sites opted out of OAI-SearchBot can still appear as navigational links. To keep content out of Google and Bing AI answers, the documented controls are noindex, nosnippet, max-snippet and data-nosnippet. Those are on-page controls, and a crawler has to be able to fetch the page to see them.

What happens if my robots.txt does not mention an AI bot? The bot falls back to the User-agent: * group if one exists. If that group allows everything, or there is no wildcard group, the bot is allowed by default. That means a file written years ago can still decide your AI search eligibility. Read the wildcard group carefully, because a Disallow: / there applies to every crawler without its own group, including search crawlers.

Can I disallow training but stay in Perplexity and Claude search? Yes. Write separate groups. Disallow ClaudeBot for training while allowing Claude-SearchBot, which Anthropic uses for search quality. Allow PerplexityBot, which Perplexity says indexes for search and is not used to crawl content for AI foundation models. Each vendor documents its tokens, so check names exactly. A typo in a user-agent line means the rule never matches.

Is Disallow enough to stop all AI fetching of my site? No. User-triggered fetchers behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. If you need a hard stop, that happens at the firewall or CDN level. Be aware the same tools can accidentally block search crawlers you meant to allow, so test each bot after any change.