What happens if my robots.txt returns an error to AI crawlers?

Crawlers that follow the Robots Exclusion Protocol (RFC 9309) read the HTTP status of /robots.txt before they read any rules. The code changes the outcome more than the file's contents do.

• 200: the crawler reads your rules and follows them per bot.
• 4xx (for example 404 or 403): the file counts as unavailable. The crawler may fetch anything.
• 5xx or a network failure (timeouts, refused connections): the file counts as unreachable. The crawler should assume the whole site is disallowed.

So a missing file is harmless, and a broken server is not. A robots.txt that times out can make a compliant crawler stop fetching your site, even if the file would have allowed it.

What this means for AI search: a retrieval crawler like OAI-SearchBot or PerplexityBot needs a clean 200, or a clean 404, to proceed. An intermittent 5xx can quietly remove eligibility. If you're not sure how your rules apply per bot, start with our per-bot robots.txt guide.

The RFC also lets crawlers use a cached copy of the file during a prolonged outage, so behavior during a long incident can vary by crawler.

Two cautions. First, the RFC describes what crawlers should do. Our evidence base does not document how each AI vendor implements every branch, as of October 2026. Second, 403 is the confusing case. By the letter of the protocol it means "no rules," but in practice it rarely comes alone.

A 403 on robots.txt is rarely a deliberate choice. It usually comes from a firewall, CDN or bot-management rule that rejects the crawler by user agent or IP. That same rule almost always blocks your content pages, so the crawler is "allowed" by the protocol and still turned away at the door.

The operators say this plainly:

• Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." Your carefully written rules for ClaudeBot or Claude-SearchBot may never be read.
• OpenAI's help center says eligibility for ChatGPT search means allowing OAI-SearchBot and confirming your host or CDN allows traffic from OpenAI's published searchbot IP addresses. robots.txt alone is not enough.

This is why robots.txt and the firewall are two separate gates. One can say yes while the other says no, and no dashboard warns you. If you suspect an IP block, read whether a blocked crawler can still read robots.txt. For diagnosing the block itself, see troubleshooting 403 errors for AI crawlers.

Answer-time fetchers may skip robots.txt

Not every AI fetcher consults the file. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user starts those actions. Perplexity says Perplexity-User "generally ignores robots.txt rules." A robots.txt error changes little for these agents. The firewall still does.

Google has no AI-only token

There is no robots.txt token that controls Google AI Overviews or AI Mode. Googlebot's access governs those surfaces, so a robots.txt outage that stops Googlebot affects Google Search and its AI features together.

CDN defaults can change without you

As of 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can start returning blocks to some AI crawlers without anyone editing a file.

Training bots and search bots are separate

An error that affects GPTBot affects training use. OAI-SearchBot controls ChatGPT search. OpenAI states each setting is independent. See how this plays out in OpenAI crawlers, robots.txt and firewalls.

You control your server, CDN and file. You don't control how each engine ranks or cites you. Fix the access layer first.

1. Request /robots.txt as each AI crawler. Use the user-agent strings each vendor documents. Record the status code, any challenge page and the response time.
2. Request a real content page the same way. A 200 on robots.txt with a 403 on your pricing page is still a block.
3. Test from more than one network. Some firewalls treat datacenter and residential traffic differently. A spoofed user agent from your laptop is not the same as the real bot's IP, so a rule that verifies bot IPs can behave differently for you than for the crawler.
4. Watch for 5xx and timeouts. An overloaded origin or a slow edge rule can turn robots.txt unreachable under load.
5. Re-test after CDN or plugin changes. Security updates are a common cause of silent regressions.

A quick manual check looks like this:

curl -sI -A "OAI-SearchBot" https://example.com/robots.txt

The robots.txt reachability playbook walks through each step. To skip the manual work, the free robots.txt reachability check or 99Visibility's AI Crawler Checker requests your page as each AI crawler and shows which ones the firewall turns away.

Is a missing robots.txt (404) bad for AI visibility? Not by itself. Under RFC 9309, a 404 means the file is unavailable, and compliant crawlers may fetch anything. You lose the ability to set per-bot rules, such as allowing OAI-SearchBot while disallowing GPTBot. If you want that control, publish a file that returns 200. If you're happy for every crawler to fetch everything, a 404 does no harm.

Does a 503 during maintenance block AI crawlers? It can. A 5xx on robots.txt makes the file unreachable, and RFC 9309 tells crawlers to assume the whole site is disallowed while that lasts. Short maintenance windows are usually tolerable, and the RFC allows cached copies during long outages. Keep robots.txt served from a reliable path, ideally the CDN edge, so it stays up when the origin does not.

If robots.txt returns 200 to a bot, am I fully accessible? No. A 200 on robots.txt proves only that the crawler can read your rules. Your content pages may still return 403, a challenge page or content that only appears after JavaScript runs. Test a representative content page with each crawler's user agent, and check that key facts like pricing are in the raw HTML.

Should I just allow AI crawlers' IP addresses in my firewall? Allowlisting can help, but don't rely on IPs alone. OpenAI publishes OAI-SearchBot IP ranges, and its help center says your host or CDN must allow them for ChatGPT search eligibility. Lists change, though, so validate on user agent plus your CDN's verified-bot programme where available, and re-test after any update.