Robots Exclusion Protocol (RFC 9309) explained

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

The Robots Exclusion Protocol (REP) is the standard that lets a site tell crawlers which URLs they may fetch, using a robots.txt file at the site root. The IETF published it as RFC 9309 in 2022, after decades as an informal convention. Crawlers pick the rule group that matches their name, then apply Allow and Disallow rules. Following those rules is voluntary. Nothing in the protocol enforces them.

Key takeaways

RFC 9309 (2022) turned the long-standing robots.txt convention into a formal IETF standard.
A crawler obeys the one group that matches its user-agent token and falls back to the * group only if no named group matches.
When Allow and Disallow rules conflict, the most specific (longest) matching path wins.
Robots.txt is a request, not access control: some user-triggered AI fetchers say they may not follow it.
For AI visibility, robots.txt rules are per bot, so blocking a training crawler and blocking a search crawler are separate decisions.

Definition

The Robots Exclusion Protocol is the set of rules a crawler reads from /robots.txt before it fetches pages on a site. The file names crawlers by user-agent token and lists the paths each one may or may not request.

The convention dates to the 1990s, but for most of its life it had no formal specification. Crawlers parsed it in slightly different ways. In 2022 the IETF published RFC 9309, which defines the file format, how crawlers match groups and rules, and how they handle errors. When people say "the REP standard", they mean RFC 9309.

This entry is a tight definition. For per-bot rules and working examples, read the robots.txt guide for AI crawlers.

How it works

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/

User-agent: *
Disallow: /private/

Group matching

A crawler looks for the group whose user-agent matches its own product token. The match ignores case (see whether AI bot names are case-sensitive).
If several groups name the same crawler, the crawler combines them.
Only if no group names the crawler does it fall back to the * group. A named group replaces the * rules entirely, so in the example above OAI-SearchBot is not bound by Disallow: /private/.

Allow and Disallow

Rules match URL paths from the start of the path.
When more than one rule matches, the longest (most specific) match wins.
If an Allow and a Disallow match with equal specificity, the RFC says the crawler should use Allow.
No matching rule means the path is allowed.

RFC 9309 also sets a minimum file size crawlers must parse, which matters for sites with very long files. That edge case is covered in what happens with a very large robots.txt.

Why it matters for AI visibility

Robots.txt is one of the few AI visibility levers with documented consequences, because AI companies publish separate tokens for separate jobs.

OpenAI's crawler page (as of October 2026) lists GPTBot for training and OAI-SearchBot for ChatGPT search. It states that "each setting is independent of the others". Disallowing OAI-SearchBot keeps a site out of ChatGPT search answers. Disallowing GPTBot only signals that content should not be used for training. Anthropic splits its crawlers the same way; see ClaudeBot, Claude-SearchBot and Claude-User.

Google is different. There is no robots.txt token that removes a page from AI Overviews or AI Mode. Googlebot rules govern all Google Search surfaces, and Google-Extended controls Gemini training and grounding without affecting Search.

The next move: read your file group by group and confirm that each retrieval bot you want (OAI-SearchBot, PerplexityBot, Claude-SearchBot) lands in a group that allows your key pages. A free robots.txt checker for AI crawlers does this per bot.

Common misconceptions

"Robots.txt blocks access." It does not. RFC 9309 says the rules are not a form of access authorization. Compliance is voluntary. Perplexity says its Perplexity-User fetcher generally ignores robots.txt rules, and OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User requests.
"If robots.txt allows a bot, the bot gets in." A firewall or CDN can still return a 403 to a crawler that robots.txt allows. Check both layers.
"Disallow removes a page from search." Google says robots.txt manages crawling, not indexing. To keep a page out of results, use noindex, which the crawler must be able to fetch to see.
"Allowing GPTBot helps ChatGPT find you." GPTBot governs training only. ChatGPT search uses OAI-SearchBot.
"llms.txt is part of the REP." It is a separate proposal that no major engine documents reading. See llms.txt vs robots.txt.

Frequently asked questions

When was RFC 9309 published?

The IETF published RFC 9309 in 2022. Before that, the Robots Exclusion Protocol existed as an informal convention from the 1990s that crawler operators interpreted on their own. The RFC wrote down the file syntax, the group and rule matching logic, caching expectations and how crawlers should treat unreachable files, so that site owners and crawler operators work from one shared text.

Does a crawler follow both its own group and the * group?

No. Under RFC 9309 a crawler obeys the group that names its user-agent token, combining multiple groups if the same name appears more than once. It uses the * group only when no group names it. If you add a group for OAI-SearchBot, repeat any wildcard rules you still want that bot to follow inside its own group.

Which rule wins if Allow and Disallow both match a URL?

The most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow rule match with the same length, RFC 9309 says the crawler should apply the Allow rule. So `Disallow: /docs/` plus `Allow: /docs/public/` lets the crawler fetch pages under /docs/public/ while blocking the rest of /docs/.

Is ignoring robots.txt illegal?

The protocol itself creates no legal obligation. RFC 9309 describes the rules as a request that crawlers are expected to honor, not an access control mechanism. Whether ignoring them has legal consequences depends on other law and contracts, which the RFC does not address. For content that must stay private, use authentication or server-level blocking, not robots.txt.

Can robots.txt keep my site out of Google AI Overviews?

Not separately from Search. Google has no robots.txt token specific to AI Overviews or AI Mode, so Googlebot rules apply to all Google Search surfaces. Google-Extended affects Gemini training and grounding but not Search inclusion. Page-level controls such as nosnippet and max-snippet are the documented way to limit what appears in Google's AI features.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.