How do Allow and Disallow rules interact for AI crawlers?

RFC 9309, the Robots Exclusion Protocol standard, resolves conflicts in two steps. Most mistakes come from mixing them up.

Step 1: pick one group

A crawler reads every User-agent line and looks for one that matches its own product token, such as OAI-SearchBot. The match is case-insensitive. If it finds one, it uses that group's rules. If several groups name the same bot, the crawler combines them into one. Only when no group names the bot does it use the User-agent: * group.

This is the part people miss. A bot-specific group replaces the wildcard group. It does not add to it. If your * group blocks /private/ and you add a group for OAI-SearchBot that only says Allow: /, OAI-SearchBot can now crawl /private/.

Step 2: the most specific rule wins

Inside the chosen group, the crawler compares the URL path against each Allow and Disallow directive. The match with the most characters wins. Line order does not matter. If an Allow and a Disallow match with the same length, RFC 9309 says crawlers should use the Allow.

So Disallow: / plus Allow: /docs/ means everything is blocked except /docs/. The longer /docs/ rule beats the one-character / rule for any path under /docs/.

Say you want ChatGPT search to reach your site but not a /private/ folder. You also want every other bot kept out of that folder.

User-agent: *
Disallow: /private/

User-agent: OAI-SearchBot
Allow: /
Disallow: /private/

Here is how OAI-SearchBot reads it. It finds its own group and ignores the * group. For /pricing, only Allow: / matches, so it may crawl. For /private/report, both rules match, but /private/ is longer than /, so the Disallow wins.

If you deleted the second Disallow: /private/ line, OAI-SearchBot would be allowed into the folder. That is the inheritance trap from the first section.

The reverse case, opening only one folder to a bot, looks like this:

User-agent: PerplexityBot
Disallow: /
Allow: /docs/

For more patterns like this, see whether you can allow an AI crawler on some pages but not others.

AI vendors run several crawlers with different jobs, and each one reads only its own group. OpenAI's crawler documentation says GPTBot governs training use, OAI-SearchBot governs ChatGPT search, and "each setting is independent of the others." You can allow OAI-SearchBot and block GPTBot in the same file. Our guide to GPTBot vs OAI-SearchBot covers which to allow.

Three consequences follow:

• A group for GPTBot does nothing for ChatGPT search. Name OAI-SearchBot if search is what you care about.
• A blanket User-agent: * with Disallow: / blocks every bot you did not name, including retrieval bots. If you want any AI search bot in, it needs its own group.
• A bot you never mention falls back to the * group. Read what happens when robots.txt doesn't mention an AI bot before assuming it is blocked or allowed.

User-triggered fetchers behave differently. As of October 2026, OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Precedence logic only matters for bots that read the file.

• Trailing slashes. Disallow: /private also matches /private-notes and /private.html. Use /private/ if you mean the folder only.
• Wildcards. RFC 9309 supports * (any sequence of characters) and $ (end of the path). Disallow: /*.pdf$ blocks PDFs. Specificity is still judged by the length of the matching rule.
• Equal-length ties. Allow: /a and Disallow: /a in one group resolve to Allow under the RFC. Avoid writing ties at all; they are hard to read and easy to break.
• Paths are case-sensitive. User-agent names are not, but /Docs/ and /docs/ are different paths.
• Duplicate groups. Two separate User-agent: OAI-SearchBot groups get merged. That can surprise you if a forgotten group lower in the file contains a Disallow.

Robots.txt also applies per host, so a subdomain needs its own file. And the file itself has to be fetchable: if a bot cannot read robots.txt, it cannot read your rules.

A correct robots.txt buys eligibility, not inclusion. OpenAI's help center states that eligibility for ChatGPT search needs two things: allowing OAI-SearchBot, and confirming your host or CDN lets traffic from OpenAI's published search bot IP addresses through. A firewall that returns 403 to a bot your file allows removes you silently. Nobody outside the vendor controls whether a crawled page is later retrieved or cited.

Here is the order to work in:

1. Write one group per AI bot you care about, and repeat every Disallow you still need inside each group.
2. Check each important URL against each bot's group using the longest-match rule. Our playbook on testing robots.txt rules for each AI bot walks through it.
3. Run the file through the free robots.txt Checker for AI Crawlers to see how each bot resolves.
4. If the file looks right but you still never appear, check the network layer. Start with what else can block you from ChatGPT when robots.txt allows AI bots.