robots.txt vs Firewall: The Two Gates for AI Crawlers

robots.txt is a text file at your site root. It names crawlers and says what each one may fetch. It enforces nothing on its own. A well-behaved bot reads it and complies.

Your CDN, web application firewall (WAF) or bot-management layer is enforcement. It sees every request and decides whether the response is a page, a 403, a rate-limit error or a challenge page. Different teams usually own the two: marketing or SEO edits robots.txt, while engineering or security sets the firewall. Neither file shows the other's settings.

OpenAI states the requirement plainly. Its help article on ChatGPT search says: "To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The operative word is "and". Fail either gate and you are out.

Passing both gates is necessary, not sufficient. Access is stage one. Whether a page is retrieved for a given question, and whether it is cited, is decided later by the engine. Nobody outside the engine controls that.

The last two rows are why reading robots.txt alone is not an audit. The file shows intent. The firewall shows outcome.

robots.txt rules are per bot, and each AI company runs several bots for different jobs. OpenAI's crawler page says GPTBot is used for training, OAI-SearchBot surfaces sites in ChatGPT search, and "Each setting is independent of the others." Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." So this is a valid policy:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

It opts out of training and stays eligible for search. The details are in whether to block GPTBot and still appear in ChatGPT search. The most common real-world error runs the other way: a blanket AI block that also removes the search bots.

robots.txt also has limits:

• User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." See PerplexityBot vs Perplexity-User for how that split works.
• Google's AI features. There is no robots.txt token for AI Overviews or AI Mode. Googlebot rules, plus noindex and snippet controls, are the levers there.
• Bots that do not comply. If a crawler ignores the file, only the firewall can stop it.

A firewall or CDN can block an AI crawler that robots.txt allows. The operators document the mechanism. How often it happens across the web has not been measured.

The usual causes are bot-protection modes, challenge pages, rate limits, country or ASN rules, and security plugins. The crawler gets one of these back:

• 403 Forbidden: the request was refused outright.
• 429: rate limited.
• A 200 with a JavaScript challenge or CAPTCHA: the crawler is automated and cannot pass it, so it gets no content.

None of this shows up in robots.txt, Search Console or your analytics. A 403 to OAI-SearchBot is a complete visibility failure for ChatGPT search that nobody on your team will see.

Two quieter variants matter. First, the firewall can block /robots.txt itself. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." Making sure crawlers can fetch robots.txt itself covers that check. Second, OpenAI publishes its crawler IP ranges as JSON files, and they list IPv4 prefixes only. An edge that answers only over IPv6 is unreachable to those crawlers. For the full list of causes, read when your CDN or firewall blocks AI crawlers.

Firewall settings can change without anyone on your side touching them. From 2026-09-15, Cloudflare sets crawlers in its Training and Agent categories to blocked by default on pages that display ads. Search crawlers remain allowed by default. The change covers new domains, new sites added to existing accounts, and all existing free-tier customers.

What it means: a site that worked in August could, as of October 2026, be refusing some AI crawlers on its ad-supported pages. robots.txt would still read exactly as before. The owner made no edit and would see nothing from outside.

The next move is a dashboard check. Open your zone's AI crawler and bot settings, see which category each crawler you care about falls into, and confirm the setting matches your robots.txt policy. Reviewing CDN bot settings for AI crawlers walks through what to look at.

Reading robots.txt tells you the policy. Testing tells you the result. Run both.

1. Parse robots.txt per agent. Resolve Allow and Disallow separately for OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot and Googlebot. Do not treat training and search bots as one.
2. Request a real content page as each crawler. Use the documented user agent string for each bot. Record the status code and whether the body is your content or a challenge page.
3. Request /robots.txt the same way. A blocked robots.txt is its own failure.
4. Vary the network if you can. A request from a datacenter and one from a residential connection can get different treatment.
5. Check your CDN or origin logs for verified bots. A test that only changes the user agent comes from your IP, not the crawler's. IP-based rules and verified-bot checks can treat it differently. Logs filtered to verified crawlers show what the real bots received. Cloudflare verifies bots through signals such as a published IP list with a stable user agent, reverse DNS, or a cryptographic signature.

On IP allowlisting: allow the operator's published ranges, but do not rely on IP alone. Pair the user agent with a verified-bot programme so spoofers cannot ride the allowlist.

The step-by-step version is in testing whether your firewall lets AI crawlers in. For a quick first pass, 99Visibility's free AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

• Use robots.txt to state policy per bot. It is the right tool for decisions like "no training, yes search." It is cheap to change and readable by everyone.
• Use the firewall to enforce against bots that ignore robots.txt or overload your servers. Accept the trade-off: blocking a search crawler at the edge removes eligibility for that engine.
• Keep the two aligned. The firewall should never refuse a bot that robots.txt allows, unless someone decided that on purpose and wrote it down.
• Test with user agents whenever robots.txt allows a bot but you never appear. If that is your situation, start with what can block you when robots.txt allows AI bots.
• Re-test after any CDN, plugin or hosting change, and after vendor default changes like Cloudflare's.
• For Google AI Overviews and AI Mode, use Googlebot access and snippet controls. Neither gate has an AI-only switch for Google.

Does blocking GPTBot remove me from ChatGPT search? No. OpenAI's crawler documentation says GPTBot governs training use and OAI-SearchBot governs ChatGPT search, and that each setting is independent of the others. You can disallow GPTBot and allow OAI-SearchBot. The reverse also holds: allowing GPTBot does nothing for search eligibility if OAI-SearchBot is blocked in robots.txt or refused at the firewall.

What does it mean when an AI crawler gets a 403? The server or an edge layer in front of it refused the request. For a search crawler, that page cannot enter the engine's index, whatever robots.txt says. The usual causes are bot-protection modes, security plugins, IP or country rules, and vendor defaults. Your analytics will not show it. Your CDN or origin logs, filtered by user agent, will.

Can I keep my site out of Google AI Overviews with robots.txt? Not with an AI-specific token, because none exists. Google's AI features use Googlebot, so blocking Googlebot blocks Search as a whole. Google says a page must be indexed and eligible to show with a snippet to appear as a supporting link. The page-level controls are noindex, nosnippet, max-snippet and data-nosnippet.

If both gates are open, will AI engines cite me? Not necessarily. Open gates make you eligible. Whether an engine retrieves your page for a given question, and whether it cites it, is decided by the engine and varies by query. Fixing access removes a block. It does not guarantee an outcome, and no one can promise one.

Should I allowlist OpenAI's published IP ranges? OpenAI's help center asks site owners to let traffic from its published searchbot IP addresses through the host or CDN. Do that, but do not rely on IP alone. The published lists contain IPv4 prefixes only, and ranges change. Combine the IP check with the user agent and a verified-bot programme in your CDN.