How to Test robots.txt Rules for Each AI Bot

Every AI company runs several crawlers, and each one reads robots.txt for its own token. A rule written for one bot says nothing about the others. OpenAI states it plainly on its crawler page: "Each setting is independent of the others."

That independence is why a single pass or fail for the whole file is useless. You need one answer per user agent, matched to what that bot actually does. As of October 2026, the tokens that matter most are:

There is no robots.txt token for AI Overviews or AI Mode. Googlebot is the control there. User-triggered fetchers also behave differently: OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Our per-bot guide to robots.txt for AI crawlers covers each token in more depth.

RFC 9309, the Robots Exclusion Protocol standard, sets the order of operations. Get this step wrong and every later check is wrong too.

• The crawler compares its product token with each User-agent line, case-insensitively.
• If one or more groups name it, it uses those groups and merges them if there are several.
• Only if no group names it does it fall back to the User-agent: * group.
• It never combines its named group with the * group.

That last rule causes most surprises. Take this file:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /private/

GPTBot may crawl /admin/, because it reads only its own group. OAI-SearchBot has no group, so it falls back to * and stays out of /admin/. If you want a rule to apply to a named bot, repeat it inside that bot's group.

The * is special only when it stands alone on the User-agent line. A line like User-agent: GPT* is not a wildcard under the standard, so do not rely on it. Write each token in full. Rules also apply per host, so each subdomain needs its own robots.txt.

Once you know the group, test each URL against its rules. RFC 9309 says the most specific match wins, meaning the rule with the longest matching path. If an Allow and a Disallow match with equal length, crawlers should use Allow. Our explainer on how Allow and Disallow interact walks through more cases.

Wildcard and path traps

• Disallow: /blog blocks /blog, /blog/ and /blog-archive, because matching is a prefix match.
• Disallow: /blog/ does not block the bare /blog URL.
• * in a path matches any run of characters, so Disallow: /*? blocks every URL with a query string.
• $ anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?v=2.
• An empty Disallow: line allows everything for that group.
• Paths are case-sensitive: /Pricing and /pricing are different rules.

File size is a quieter trap. The standard only requires crawlers to parse at least 500 kibibytes, so rules past that point may never be read. See whether a very large robots.txt can break AI crawler rules.

1. Fetch https://yourdomain.com/robots.txt directly and note the HTTP status code. Repeat for every subdomain that serves content.
2. List the URLs that carry your load-bearing facts: homepage, pricing, product pages, docs, about page.
3. For each token in the table above, find its group: a named group if one exists, otherwise *.
4. Merge any duplicate groups for the same token.
5. For each URL, find every Allow and Disallow rule in that group that matches. Keep the longest. Break ties toward Allow.
6. Write down the verdict per bot per URL in a simple grid.
7. Compare each verdict with your intent. Ask whether you meant to block training, search, or both.
8. Fix mismatches, redeploy, and rerun steps 1 to 6.
9. Run a reachability test for every bot your file allows (see below).

If you would rather not resolve rules by hand, the free robots.txt checker for AI crawlers does steps 3 to 6 for you.

• robots.txt returns 200 on every host, not a redirect loop, 5xx or timeout.
• OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Googlebot and bingbot resolve to the access you intend.
• Training bots (GPTBot, ClaudeBot, Google-Extended) resolve to the policy you chose, separately from search bots.
• No blanket User-agent: * / Disallow: / sits above an Allow you assumed would apply.
• Rules from the * group that you need for named bots are repeated inside those groups.
• Path rules use the intended trailing slash, case and $ anchor.
• The file is well under 500 KiB.
• Pricing and product URLs are allowed for every search bot you want.
• A reachability test confirms each allowed bot actually receives a 200.

Good

Each search bot resolves to Allow on your key URLs. Training bots resolve to whatever you decided on purpose. The file returns 200 and parses cleanly. That buys eligibility, not citations: allowing a bot lets it fetch your pages, and nothing more is promised.

Bad

• A blanket AI block. A file that disallows every AI token at once often removes ChatGPT search eligibility by accident, because it catches OAI-SearchBot along with GPTBot.
• A named group that drops your * rules. You added a group for one bot and lost protections you thought applied.
• A 5xx on robots.txt. Under RFC 9309, crawlers must treat an unreachable file as a full disallow.
• A 4xx on robots.txt. Crawlers may then treat the site as fully allowed, which may not be what you want either.
• A rule past the parse limit. It looks correct in the file and has no effect.

Fix the bad results first, then move on to network access.

robots.txt is a request, not a gate. Your firewall, CDN or bot-management settings decide what a crawler actually receives. A bot your file allows can still get a 403 or a CAPTCHA, and that failure is silent.

OpenAI's help center makes the point directly: to be eligible, "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The word is and, not or. OpenAI publishes those ranges at openai.com/searchbot.json. Anthropic also warns that IP blocking can stop it reading your robots.txt at all.

CDN defaults can change without you editing anything. Since 2026-09-15, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts, and existing free-tier customers.

So after the file passes:

• Check that each bot can fetch robots.txt itself.
• Request a real content URL as each crawler and record the status code. The free firewall test for AI bots does this, as does the AI Crawler Checker, which shows which crawlers your firewall turns away.
• Treat any 403, 429 or challenge page as a fail, even if robots.txt says Allow.

Rerun the full test whenever one of these changes:

• robots.txt itself, including CMS or plugin updates that rewrite it.
• Your CDN, WAF or bot-management rules, or your hosting plan.
• A new subdomain, site section or migration.
• A vendor announces a new crawler token or changes what an existing one does.

Outside those events, run it monthly. Crawlers cache robots.txt, and RFC 9309 says they should not use a cached copy for more than 24 hours unless the file is unreachable. Expect fixes to take effect within about a day, then confirm with a fresh reachability test. If you need a clean starting point, adapt the robots.txt template for AI search and training bots.