Is there a tool that tests whether my firewall is blocking AI crawlers?

Access to an AI engine runs through two gates. robots.txt says which bots may crawl. Your web application firewall, CDN or bot-management layer decides which requests actually get through. The two are configured by different people and often disagree. The difference between the two gates is why a robots.txt check alone can't answer this question.

OpenAI states the conjunction plainly on its ChatGPT search help page: to be eligible, allow OAI-SearchBot to crawl the site and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Its advertiser guidance adds that bot-protection systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."

A test worth using does four things for each crawler:

1. Requests a representative content page (not just the homepage) with the crawler's documented user-agent string.
2. Records the result: 200 is a pass; 403 is a block; 429 is rate limiting; an HTML page containing a JavaScript challenge or CAPTCHA is a fail, because automated crawlers can't solve it.
3. Fetches /robots.txt with the same user agent, since a blocked robots.txt is its own failure.
4. Compares each result with what robots.txt allows for that bot, and flags every disagreement.

The free AI Crawler Checker on 99visibility.com does this: it requests a page as each AI crawler and shows which ones the firewall turns away.

robots.txt rules are per bot, and so are the consequences. Test the retrieval bots first, because blocking them removes you from live AI answers:

• OAI-SearchBot: OpenAI says sites opted out of it "will not be shown in ChatGPT search answers."
• PerplexityBot: Perplexity's search indexing crawler.
• Claude-SearchBot: Anthropic's search crawler, separate from ClaudeBot (training).
• Meta-WebIndexer: Meta states allowing it "helps us cite and link to your content in Meta AI's responses."
• Googlebot and bingbot: index eligibility for Google's AI features and for Copilot.

Training bots (GPTBot, ClaudeBot, Google-Extended) are a separate decision. OpenAI says each setting "is independent of the others," so blocking GPTBot doesn't remove you from ChatGPT search, and allowing it doesn't put you there.

The robots.txt fetch matters more than it looks. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file," which means your rules may never be read at all. A robots.txt reachability check covers this part on its own.

Every third-party test has the same limit: it sends the crawler's user-agent string, but not from the crawler's IP addresses and without its verification. Your firewall may react to that in either direction.

• False pass. Some bot-management rules only challenge traffic that looks like a bot from a datacenter. A test from a different network may get a 200 that the real crawler would not.
• False fail. A firewall that verifies bots may block a spoofed OAI-SearchBot as an impostor while letting the real one through. That is the firewall working correctly.

Cloudflare's documentation describes verified bots as those identified by a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. OpenAI publishes its searchbot IP ranges in a JSON file but also warns against relying on IP allowlisting alone. Good practice is to verify on user agent plus a verified-bot programme.

This is the main reason crawler checkers give different results for the same site. Run the test from more than one network (a datacenter and a residential connection) and treat a disagreement as a reason to check logs, not as a verdict.

If you want to reproduce a single request yourself, this is enough:

curl -I -A "<user-agent string from the vendor's bots page>" https://example.com/pricing

Your origin or CDN logs record what the real crawlers received. Filter them to verified requests from OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer and bingbot, then look at three things:

• Status codes per bot. Repeated 403s or 429s to a verified retrieval bot are a confirmed block.
• Challenge events. Your bot-management dashboard usually logs these separately from status codes.
• Absence. A bot your robots.txt allows that never appears may be blocked upstream, or may simply not have crawled yet. Absence alone doesn't prove which.

No outside tool can read your logs, so this step is yours or your hosting provider's. The step-by-step firewall testing playbook walks through the log filters.

Nobody has published how often a firewall blocks an AI crawler that robots.txt allows. The mechanism is documented by the operators; the frequency is not measured. These are the settings to check first:

• CDN defaults that changed without your input. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers remain allowed by default. A site can change status without anyone editing it.
• Blanket "block AI bots" toggles that sweep up retrieval bots along with training bots.
• Security plugins and rate limits on WordPress and similar platforms; see whether WordPress security plugins block AI crawlers.
• Country or ASN blocks that catch crawler IP ranges.

The guide to CDN and firewall blocks covers the fix for each.

1. Run a per-crawler test on two or three important pages, including pricing.
2. For every bot where robots.txt says allow but the test shows 403, 429 or a challenge, confirm in your logs.
3. Add an allow rule for verified retrieval bots in your WAF or CDN, then retest.
4. Recheck after any CDN plan change or security update.

Be clear about the outcome. Fixing access makes you eligible to be fetched. It doesn't control whether an engine retrieves or cites you for a given question. That part sits with the engine.

Does blocking GPTBot stop ChatGPT from showing my site? No. GPTBot governs training use only. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent of the others. You can block GPTBot and still appear in ChatGPT search, as long as OAI-SearchBot is allowed in robots.txt and your firewall lets it through. Check both crawlers separately, because a blanket AI block often catches OAI-SearchBot by accident.

Can I just allowlist OpenAI's IP addresses? OpenAI publishes its searchbot IP ranges in a JSON file, and its help page asks you to confirm your host or CDN allows them. It also warns against IP-only allowlisting. A safer setup checks the user agent and uses your CDN's verified-bot programme, so impostors using a crawler's name are still filtered while the real crawler gets through.

If my test shows a 200, am I definitely fine? Not definitely. A spoofed request can be treated differently from the verified crawler, so a 200 from a test tool is strong evidence, not proof. Also check that the content that matters is in the raw HTML, since most AI crawlers don't run JavaScript. Then confirm in your server logs that the real bots receive 200s.

Do user-triggered fetchers follow the same rules? Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Your firewall can still block them. Index-time crawlers and answer-time fetchers are different controls, so test both if you care about pages being read when a user asks a question in the moment.