Making Sure Crawlers Can Fetch robots.txt Itself

Your robots.txt is a set of instructions that each crawler downloads and reads before it fetches anything else. If the download fails, the instructions don't arrive. The crawler then falls back on its own defaults, and those defaults may not match what you wrote.

This is easy to miss because the file looks fine from a browser. You open /robots.txt, see your rules, and move on. But a crawler doesn't arrive like a browser. It comes from a datacenter IP range with a bot user agent, and that's exactly the traffic firewalls and bot-management tools are built to challenge.

Anthropic says this directly in its crawler documentation: blocking its IP addresses "impedes our ability to read your robots.txt file." In other words, a network block doesn't just stop page fetches. It also stops the bot from learning what you allowed and what you didn't.

OpenAI describes the same two-part gate for ChatGPT search. To be eligible, a site must allow OAI-SearchBot in robots.txt and confirm that its host or CDN allows traffic from OpenAI's published searchbot IP addresses. A correct robots.txt behind a blocking firewall fails the second half. The two gates for AI crawlers work independently, so you have to check both.

The response code a crawler gets on /robots.txt decides what it does next. RFC 9309, the Robots Exclusion Protocol standard, sets the defaults. Each operator decides how closely its crawler follows them.

A 4xx response, including 403

RFC 9309 treats a 4xx on robots.txt as "unavailable" and lets the crawler access any resources. So a 403 from your firewall doesn't block the bot by itself. It just erases your rules. Any Disallow you wrote for a training crawler goes unread. If the firewall also returns 403 on your pages, the bot gets nothing at all. OpenAI's own guidance notes that bot-protection systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."

A 5xx response or timeout

RFC 9309 treats a server error as "unreachable" and tells the crawler to assume a complete disallow. A robots.txt that times out or returns 503 to bots can quietly shut a crawler out of the whole site, even if your rules say allow.

A 200 that isn't really robots.txt

Some challenge pages return HTTP 200 with an HTML body: a CAPTCHA, a JavaScript check or a "verifying your browser" screen. The status code looks healthy. The body contains no valid rules. OpenAI notes that because its crawlers are automated, these checks may block access. What a given crawler does with an HTML body is up to its operator, but none of them can read rules that aren't there.

For a deeper look at each failure, see what happens when robots.txt returns an error to AI crawlers and whether a bot can still read robots.txt after an IP block.

This is a request-and-classify test. You fetch one URL, /robots.txt, as each crawler and record exactly what comes back.

1. List the crawlers you care about. At minimum: OAI-SearchBot and GPTBot (OpenAI), Claude-SearchBot and ClaudeBot (Anthropic), PerplexityBot, Meta-WebIndexer, Googlebot and bingbot. Copy each full user-agent string from the operator's own documentation.
2. Request /robots.txt with each user agent. A command-line request works:

curl -sS -o robots.out -w "%{http_code} %{content_type}\n" -A "OAI-SearchBot" https://example.com/robots.txt

Replace the -A value with the full string for each bot. 3. Run it from two places. Test from a datacenter server (a cloud VM) and from a residential connection. Firewalls often treat datacenter traffic differently, and real crawlers come from datacenters. 4. Record four things per bot. The status code, the content type, any redirect chain, and the first lines of the body. 5. Open the body. Confirm it starts with User-agent: lines and contains your actual rules, not HTML. 6. Compare against a plain browser fetch. If the browser gets your file and a bot user agent gets something else, your edge is treating bots differently. 7. Confirm with logs. A spoofed user agent from your own IP isn't the real crawler. Some firewalls block fake bots harder than real ones, or the reverse. Filter your server or CDN logs to requests for /robots.txt from each bot's published IP ranges (OpenAI publishes OAI-SearchBot's at openai.com/searchbot.json) and check the status codes those requests received.

Step 7 is the only one that shows what the real crawler saw. Steps 1 to 6 tell you where to look.

If you'd rather not script it, the free robots.txt reachability check runs the request side for you. 99Visibility's AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

Classify each bot's response into one of these buckets.

Two patterns deserve extra attention. The first is a pass from your residential connection and a 403 from the datacenter. That's a network-layer block, and it's invisible when you check from your office. The second is a pass for Googlebot and bingbot with a failure for the AI crawlers. That usually means a bot-management category is set to block AI traffic while leaving search engines alone.

Run through this list for every host you publish on, including subdomains, since each serves its own robots.txt.

• /robots.txt returns 200 to every AI crawler on your list
• The content type is text/plain, not text/html
• The body starts with your real rules, with no challenge or login page
• The result is the same from a datacenter IP and a residential IP
• No redirect loop, and any redirect lands on a 200
• Rate limiting does not return 429 to crawlers on /robots.txt
• Bot-management or WAF rules exempt /robots.txt, or allow verified bots
• Security plugins are configured not to block crawler user agents on /robots.txt (see whether WordPress security plugins block AI crawlers)
• Server or CDN logs show 200s on /robots.txt for requests from published crawler IP ranges
• If you use Cloudflare, you know which AI crawler categories your zone blocks

Passing this list means your stated policy can reach each crawler. It doesn't mean any engine will cite you. Reachability is an access condition, and what the models do with your pages after that is outside your control.

robots.txt doesn't govern every AI fetch, even when it's reachable. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because those requests are started by a user. Perplexity says Perplexity-User "generally ignores robots.txt rules." So a reachable robots.txt controls index-time crawlers like OAI-SearchBot and PerplexityBot, not every user-triggered fetch. The guides on OpenAI crawlers and firewalls and PerplexityBot vs Perplexity-User cover which bot follows which rule.

If you want to block a user-triggered fetcher, robots.txt alone won't do it reliably. If you want to allow one, the firewall has to let it through, and robots.txt is beside the point.

robots.txt reachability changes without anyone editing robots.txt. A firewall rule, a CDN setting, a hosting migration or a plugin update can flip a pass to a 403 overnight.

A sensible schedule:

• After any change to your CDN, WAF, bot-management settings, hosting, or security plugins
• After a migration or redesign, including new subdomains
• Monthly as a routine check, because vendors change defaults too

That last point is real, not theoretical. From 15 September 2026, Cloudflare began blocking Training and Agent crawler categories by default on pages that show ads, for new domains, new sites on existing accounts, and existing free-tier customers. A site on that path could lose AI crawler access without its owner touching anything. If your last check predates that date and you use Cloudflare, run it again.

Does a 403 on robots.txt block the crawler from my site? Not by itself. RFC 9309 treats a 4xx on robots.txt as unavailable, which lets the crawler proceed as if there were no rules. Your disallows go unread. If the same firewall rule also returns 403 on your pages, the crawler gets nothing. Either way, your stated policy has stopped doing its job, so fix the block rather than relying on it.

Is spoofing a crawler's user agent a reliable test? It's a good first pass, not proof. A request from your own IP with a bot user agent shows how your edge treats that string. It doesn't show how it treats the real crawler from its real IP range, and some firewalls handle fake bots differently. Confirm by checking your server or CDN logs for requests to /robots.txt from each operator's published IP ranges.

Should I allowlist crawler IP addresses? You can use published ranges, such as OpenAI's searchbot.json, to confirm which log entries are real crawlers. Relying on IP lists alone is fragile because they change, and OpenAI itself warns against IP-only allowlisting. Pair the published ranges with user-agent checks or your CDN's verified-bot feature, and exempt /robots.txt from challenges so every crawler can read your rules.

My robots.txt returns 200 but the page shows a CAPTCHA. Is that a pass? No. The status code says success, but the body is HTML with no valid rules in it. The crawler can't read a policy that isn't there, and how each operator handles that body is its own choice. Treat any 200 with an HTML challenge body as a failure, and turn the challenge off for verified bots on /robots.txt.

If every crawler can read my robots.txt, will AI engines cite me? Reachability doesn't guarantee citations. It removes one block. After that, each engine decides whether to crawl your pages, whether they enter its index, and whether an answer uses them. You control access: robots.txt, firewall rules, rendering and snippet controls. You don't control what the models say. A clean reachability result means your policy reaches the crawler, nothing more.