Are AI bot names in robots.txt case-sensitive?

The Robots Exclusion Protocol, standardised as RFC 9309, says crawlers must use case-insensitive matching to find the group that matches their product token. So User-agent: gptbot, User-agent: GPTBot and User-agent: GPTBOT all address OpenAI's training crawler.

The rest of the line is less forgiving. RFC 9309 treats the path in an Allow or Disallow rule as case-sensitive. Disallow: /Pricing does not block /pricing.

The real risk is spelling. A crawler only reads a group whose token matches its own. If you write OAISearchBot or Claude-Search-Bot, the group is valid syntax that nobody obeys. Nothing warns you. The bot falls through to the * group, or to no rules at all.

What this means for you: stop worrying about capital letters and start checking every token character by character against the vendor's page.

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines followed by rules. A crawler reads the file like this:

1. It looks for a group whose token matches its own product token, ignoring case.
2. If several groups match, it combines their rules.
3. If none match, it uses the * group.
4. Within the chosen group, the most specific matching path wins.

Step 3 catches people out. Once a crawler finds a group that names it, it ignores the * group entirely. Say you block /admin/ under * and add a separate GPTBot group that only allows /blog/. GPTBot no longer sees your /admin/ rule. Repeat shared rules inside every named group.

The user agent token is also not the full HTTP user-agent string. RFC 9309 expects the short product token, which contains only letters, underscores and hyphens. Pasting a full browser-style string or a version number into the User-agent line produces a group that matches nothing.

If a bot isn't named anywhere, the answer to whether an unlisted AI bot is allowed depends on your * group.

These are the errors worth checking first. Each one leaves a file that looks fine to a human and does nothing for the bot.

• Wrong spelling or punctuation. OAI-SearchBot has a hyphen. Claude-SearchBot and Meta-WebIndexer do too. Dropping or moving it breaks the match.
• Comma-separated names. User-agent: GPTBot, ClaudeBot is not a list. Put each token on its own User-agent line above the shared rules.
• Confusing training bots with search bots. OpenAI documents that GPTBot governs training use and OAI-SearchBot governs ChatGPT search, and states that each setting is independent. A blanket block written for training can remove search eligibility if you also list the search bot.
• Missing colon or a typo in the field name. Useragent or User agent without the hyphen is not a recognised field.
• Path case mismatches. Your rules must match the case of your real URLs.
• Rules split from their group. A stray User-agent line between rules starts a new group, which changes who the rules apply to.

File size can also cause trouble. See whether a very large robots.txt can break AI crawler rules if yours runs to thousands of lines.

Copy tokens from each vendor's own documentation, not from a blog post or a plugin's preset list. As of October 2026, the documented names include:

The user-triggered fetchers behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. There's also no robots.txt token for Google AI Overviews or AI Mode: Googlebot rules are the control there.

Our digests of OpenAI's crawler documentation and Anthropic's crawler documentation quote the vendor wording for each bot.

A clean example that allows search bots and blocks two training bots:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

A correct file is necessary but not sufficient. Check three things in order.

1. Parse the file per bot. Resolve which group each named crawler selects and what it may fetch. The robots.txt Checker for AI Crawlers does this for the main AI bots.
2. Confirm robots.txt itself is reachable. Anthropic warns that IP blocking impedes its ability to read your robots.txt file.
3. Request a page as each crawler. A firewall or CDN can return a 403 to a bot your file allows. OpenAI's help centre says eligibility for ChatGPT search needs both OAI-SearchBot allowed and your host or CDN allowing its published IP addresses. Our page on OpenAI crawlers, robots.txt and firewalls covers that second gate.

What stays outside your control: allowing a bot buys eligibility only. It does not make any engine retrieve, cite or rank your pages. What you control is not blocking them by accident.