Testing Whether Your Firewall Lets AI Crawlers In

99Visibility Research · Updated October 21, 2026 · 7 min read

Short answer

To test whether your firewall blocks AI crawlers, request a real content page and your robots.txt with each crawler's documented user-agent, then classify the response. A 200 that returns your real content passes. A 403, a repeated 429, or a challenge page fails. OpenAI says ChatGPT search eligibility needs both a robots.txt allow for OAI-SearchBot and a host or CDN that accepts OpenAI's published searchbot IP addresses.

Key takeaways

robots.txt and your firewall are two separate gates, and OpenAI requires a site to pass both for ChatGPT search eligibility.
Test each retrieval crawler separately, from two networks, against real content pages and robots.txt itself.
A 200 only counts as a pass when the body matches what a browser sees; a challenge page served with a 200 is a fail.
A spoofed user-agent test finds suspect rules, but your server or CDN logs for verified bots are the ground truth.
How often firewalls block AI crawlers across the web has not been measured, so check your own site after every security or CDN change.

Why robots.txt is only half the test

robots.txt tells a crawler what it may fetch. Your firewall, CDN or bot-management layer decides whether the request gets through at all. These are separate systems, and they can disagree without anyone noticing.

OpenAI states the rule in its help article on ChatGPT search (checked August 2026): to make a site eligible, "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The key word is "and". A robots.txt allow does nothing if a web application firewall answers OAI-SearchBot with a 403.

The failure is silent. Nothing in robots.txt, your CMS or your analytics flags it. Crawler operators document the mechanism, but nobody has published how often it happens across the web. Treat it as something to check on your own site, not a known rate. The robots.txt vs firewall comparison lays out both gates side by side.

Defaults can also change under you. From 15 September 2026, Cloudflare set its Training and Agent crawler categories to blocked by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers stayed allowed by default. A site can change its AI access without its owner editing anything, which is why CDN and firewall blocking belongs in every access review.

Which AI crawlers to test

Decide your intended policy first, then test against it. Training crawlers and search crawlers do different jobs, and OpenAI says "Each setting is independent of the others."

OAI-SearchBot: ChatGPT search. OpenAI says sites opted out "will not be shown in ChatGPT search answers".
GPTBot: training only. Blocking it does not remove you from ChatGPT search, as covered in whether to block GPTBot.
ChatGPT-User: fetches a user triggers. OpenAI says robots.txt rules "may not apply" to it, so your firewall is the real control.
PerplexityBot: the Perplexity search index. Perplexity says Perplexity-User "generally ignores robots.txt rules".
Claude-SearchBot and ClaudeBot: Anthropic separates search quality from training, with Claude-User for user-triggered fetches.
Meta-WebIndexer: Meta says allowing it "helps us cite and link to your content in Meta AI's responses."
bingbot: the Bing index backs Microsoft Copilot.

Googlebot is worth one request too. Google has no separate robots.txt token for AI Overviews or AI Mode, so a Googlebot block affects those surfaces as well.

Run the reachability test step by step

Pick three URLs. Your homepage, the page that carries your most important facts (pricing or the main product page), and /robots.txt.
Record a browser baseline. Load each URL normally. Note the status code, rough page size, and one exact phrase that only appears in the real content.
Copy the full user-agent strings. Take them from each operator's own documentation (OpenAI, Anthropic, Perplexity, Meta). Do not guess them.
Request each URL with each user-agent. A short curl command is enough:

curl -s -o page.html -w "%{http_code} %{size_download}\n" \
  -A "PASTE_FULL_USER_AGENT_STRING" \
  https://www.example.com/pricing
grep -c "your known phrase" page.html

Run it from two networks. Use a cloud server and an office or home connection. Bot management often scores datacenter traffic differently from residential traffic.
Keep redirects visible. curl does not follow redirects by default, so a 301 or 302 to a challenge URL shows up instead of hiding.
Compare against your policy. Every crawler you meant to allow should match the browser baseline. Every crawler you meant to block should be blocked where you chose to block it.
Confirm in your logs. Check what the real crawlers received (see the limits section below).

How to read 200, 403, 429 and challenge results

Result	What it usually means	Verdict
200, body contains your phrase and size is close to baseline	The request reached your content	Pass
200, body is a "checking your browser" page or CAPTCHA	Bot management intercepted the request	Fail
403 Forbidden	A firewall or bot rule refused the crawler	Fail
429 Too Many Requests	Rate limiting	Fail if it repeats at a slow request rate
503 with challenge markup	A challenge served as an error	Fail
301 or 302 to a challenge or login URL	Interception by redirect	Fail
Timeout or connection reset	A network-level block	Fail

What good looks like

Every retrieval crawler you intend to allow gets a 200 with the same content a browser sees, from both networks, on all three URLs. robots.txt itself returns 200 to every crawler.

What bad looks like

A 403 to OAI-SearchBot on your pricing page is a complete, silent failure for ChatGPT search on that page. A blocked /robots.txt is its own problem: Anthropic warns that IP blocking "impedes our ability to read your robots.txt file", so the crawler cannot even see your rules. A 200 challenge page is the easiest to miss, because the status code looks healthy.

What a spoofed request cannot tell you

Your curl request claims to be OAI-SearchBot, but it comes from your IP, not OpenAI's. Good bot management checks more than the user-agent: published IP ranges, reverse DNS or verified-bot programs. That cuts both ways.

Your spoofed request may be blocked while the real crawler gets through. That is a rule to inspect, not proof of a block.
A rule that trusts the user-agent alone may pass your test and also let impostors in.

The ground truth is your server or CDN logs, filtered to verified crawler traffic. Look for the status codes the real OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer received. The free AI bot log check walks through that.

Check IP versions too. OpenAI publishes its searchbot ranges at openai.com/searchbot.json, and as of August 2026 those files listed IPv4 prefixes only. A site reachable only over IPv6 cannot be reached by those crawlers. OpenAI also warns against relying on IP lists alone, so read whether to allowlist OpenAI's IPs before writing an IP-only rule.

If you would rather not script this, 99Visibility's AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

Firewall and CDN access checklist

Written policy: which training crawlers and which search crawlers you allow.
robots.txt matches that policy, per bot.
Homepage, key facts page and /robots.txt tested with each crawler's documented user-agent.
Tests run from a datacenter network and a residential network.
Every allowed crawler gets a 200 with real content, not a challenge page.
No 403, repeated 429, challenge redirect or timeout for allowed crawlers.
CDN bot categories (Search, Agent, Training) reviewed in your CDN dashboard.
Security plugins and managed WAF rule sets reviewed for bot blocking.
Origin reachable over IPv4.
Logs show verified crawlers receiving 200s on the pages that matter.
Results saved with a date, so the next test has a baseline.

How often to repeat the test

Run it whenever something in the request path changes:

A new CDN, WAF rule set or bot-management setting.
A new security plugin or plugin update (see how WordPress security plugins treat AI crawlers).
A hosting, DNS or IPv6 change.
A vendor default change, like Cloudflare's on 15 September 2026.

Outside those events, a monthly run is a sensible baseline. That cadence is our judgment, not a measured threshold. Keep the saved results, because a change between two dated runs points straight at the cause.

Passing this test buys eligibility, nothing more. Whether an engine retrieves and cites your page is up to the engine. What you control is making sure your firewall is not the reason it can't.

Frequently asked questions

Should I allowlist OpenAI's crawler IP ranges?

OpenAI publishes IP ranges for its crawlers and asks hosts and CDNs to allow its searchbot addresses. It also warns against IP-only allowlisting, because ranges change. A safer rule combines the documented user-agent with verified-bot checks your CDN offers. After any change, rerun the reachability test and confirm in your logs that the real OAI-SearchBot receives 200 responses on the pages you care about.

If every crawler gets a 200, will AI engines cite me?

No. Access makes a page eligible, not chosen. Each engine decides per query whether to search at all, which pages to retrieve and which to cite, and none of that is in your control. What the test removes is one specific, silent reason for absence. Other blocks still matter, such as content that only appears after JavaScript runs, which most AI crawlers do not execute.

Do I need to test GPTBot if I only care about ChatGPT search?

Not for search. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the search crawler, with independent settings. Test GPTBot only if you have a training policy you want to confirm. If you block training, make sure the rule targets GPTBot specifically and does not also catch OAI-SearchBot, which is a common side effect of blanket AI blocks.

Can robots.txt alone stop user-triggered AI fetches?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules. For these fetchers, your firewall or CDN is the effective control. That is one more reason to know exactly what your firewall does to each AI user-agent, whether your goal is to allow them or to block them.

Why does a 200 response sometimes count as a failure?

Some bot-management systems serve a challenge page, such as a browser check or CAPTCHA, with a 200 status. The crawler receives a page, but not your content. That is why the test compares the response body against a browser baseline: check for a known phrase and a similar page size. A small 200 response without your phrase is an interception, not a pass.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.