If robots.txt doesn't mention an AI bot, is it allowed?

An AI crawler that your robots.txt does not mention is allowed, unless your wildcard group says otherwise. That follows directly from how RFC 9309, the Robots Exclusion Protocol, tells crawlers to pick their rules.

A compliant crawler reads the file and looks for a group to follow, in this order:

1. A group naming its own user agent. If a group says User-agent: OAI-SearchBot, OAI-SearchBot follows that group.
2. The wildcard group. If no group names it, the crawler follows the group under User-agent: *.
3. Nothing. If no group names it and there is no wildcard group, no rules apply.

Inside whichever group applies, the default is allow. A path is off limits only if a Disallow rule matches it. An empty file, a missing file, or a file that only talks to Googlebot leaves an unlisted AI bot free to crawl.

What this means: your real policy for every AI bot you forgot to name is whatever your wildcard group says. Read that group first. For how individual rules resolve once a group is chosen, see how Allow and Disallow rules interact.

The common mistake is a blanket rule meant to stop AI training:

User-agent: *
Disallow: /

This blocks every compliant crawler that is not named elsewhere in the file. That includes the retrieval bots, not just the training bots. As of October 2026, the vendors document these roles:

• GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot is separate and controls ChatGPT search. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• PerplexityBot indexes for Perplexity search. Perplexity says it is not used to crawl content for AI foundation models.
• ClaudeBot collects training data for Anthropic. Claude-SearchBot is a separate agent tied to Claude's search.

OpenAI states that each of its settings is independent of the others. A wildcard disallow does not respect that split. It removes ChatGPT search eligibility along with training use, and it also blocks Googlebot and Bingbot if they are not named. If that outcome surprises you, read whether blocking all AI bots hides you from AI search.

If your goal is to opt out of training while staying eligible for AI search, name the training bots and leave the wildcard open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Here GPTBot, ClaudeBot and Google-Extended each match their own group and are blocked. OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and every other unnamed crawler fall to the wildcard group and are allowed. Note that Google-Extended also cuts grounding for Gemini Apps, per Google's crawler documentation, and Google says it does not affect inclusion in Google Search.

If you prefer a default-deny file, you must name every bot you want in, and Googlebot and Bingbot belong on that list. Missing one is a silent exclusion. The per-bot robots.txt guide lists the user agents for each platform, and the GPTBot question covers the OpenAI split in detail.

A named group replaces the wildcard

A bot that matches a named group follows only that group. It does not also inherit the wildcard rules. If you write a GPTBot group with one Disallow: /private/ line, GPTBot ignores your User-agent: * rules completely. Copy any wildcard rules you still want into each named group.

Names match without regard to case

RFC 9309 tells crawlers to match the product token case-insensitively. Spelling still matters. More on that in are AI bot names case-sensitive.

An unreachable robots.txt

Under RFC 9309, if robots.txt returns a 4xx error, crawlers may treat the site as unrestricted. If it returns a 5xx or cannot be reached, crawlers should assume a complete disallow. A flaky server can block every bot without any rule changing.

User-triggered fetchers

OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. See whether all AI crawlers obey robots.txt.

Google's AI features have no separate token

There is no robots.txt user agent for AI Overviews or AI Mode. Googlebot rules are the control there.

robots.txt is one gate. OpenAI's help page says eligibility for ChatGPT search also needs your host or CDN to allow traffic from OpenAI's published searchbot IP addresses. A file that allows OAI-SearchBot does nothing if your firewall returns a 403. The OpenAI crawlers and firewalls page explains that second gate.

A short checklist:

1. Read your User-agent: * group and confirm it says what you want for every bot you have not named.
2. List the search bots you want in (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) and check none is caught by a wildcard disallow.
3. Check each named group carries every rule it needs, since it will not inherit the wildcard.
4. Request a page as each crawler to confirm the network layer agrees with the file. The AI Crawler Checker does this and shows which crawlers the firewall turns away.

Allowing a bot buys eligibility only. It does not guarantee retrieval, citation or traffic. Removing a block is the part you control.

Does an empty robots.txt allow every AI crawler? Yes. Under RFC 9309, if there are no groups at all, no rules apply, and a compliant crawler may fetch any path. The same holds if robots.txt returns a 4xx error such as 404. An empty or missing file is effectively an allow-all for every bot, including training crawlers like GPTBot and ClaudeBot. If you want to opt out of training, you have to write that rule explicitly.

If I name a bot and allow one folder, can it crawl everything else? It can crawl everything its named group does not disallow. A named group replaces the wildcard group for that bot, so rules under User-agent: * no longer apply to it. If your wildcard group disallows /admin/ and your GPTBot group only allows /blog/, GPTBot is not blocked from /admin/. Repeat every rule you want inside each named group.

Will blocking GPTBot remove me from ChatGPT search? No. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the crawler for ChatGPT search, and says each setting is independent. You can disallow GPTBot and allow OAI-SearchBot in the same file. The risk comes from a wildcard disallow that catches OAI-SearchBot because it is not named. That removes ChatGPT search eligibility even though you only meant to stop training.

Does robots.txt stop a crawler that ignores it? No. robots.txt is a request that compliant crawlers honour, not an access control. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says its rules may not apply to ChatGPT-User because a person triggers the fetch. If you need to enforce a block, that happens at your server, CDN or firewall. The same layer can also block bots your file allows, so check both.