Free Test: Is Your Firewall or CDN Blocking AI Bots?

The test answers one question: when an AI crawler asks for your page, does your server hand it over? It does not check what the AI says about you, or whether you get cited.

This matters because robots.txt is only a request. A web application firewall, a CDN bot rule or a security plugin sits in front of your site and makes its own decision. If it returns 403 Forbidden, a CAPTCHA or a JavaScript challenge, the crawler gets nothing, whatever robots.txt says.

OpenAI states the conjunction plainly. To make a site eligible for ChatGPT search, you allow OAI-SearchBot to crawl and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Both conditions, not one. The two gates compared side by side shows why passing the first tells you nothing about the second.

Three things cause most silent blocks.

• Bot management rules. WAFs and CDNs score traffic and challenge anything that looks automated. OpenAI's own guidance notes these systems can mistakenly block legitimate crawlers, often with 403 errors, and that challenge checks can block its automated crawlers.
• Security plugins. On WordPress, firewall and anti-scraping plugins often block unfamiliar user agents by default. See whether WordPress security plugins block AI crawlers.
• Changed defaults. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads. That applies to new domains, new sites added to existing accounts and all existing free-tier customers. Search crawlers stay allowed by default. Nobody on your team has to touch a setting for access to change.

The guide to CDN and firewall blocks covers each vendor's controls in more depth. The point here: your robots.txt can be perfect and the crawler can still be turned away.

You need a terminal with curl. Replace the example URL with a page that matters, such as your pricing or product page.

1. Get a baseline as a browser. This shows what a normal visitor gets.

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://www.example.com/pricing

2. Copy the exact crawler user-agent strings from each vendor's documentation: OpenAI's crawler overview (OAI-SearchBot, GPTBot, ChatGPT-User), Perplexity's crawler page (PerplexityBot, Perplexity-User) and Anthropic's crawler article (ClaudeBot, Claude-SearchBot). Use the full string, since some rules match on more than the bot name.

3. Request the page as each crawler.

curl -s -o /dev/null -w "%{http_code}\n" -A "PASTE-FULL-OAI-SEARCHBOT-STRING" https://www.example.com/pricing

4. Look at the body, not just the code. Some challenges return 200 with a challenge page.

curl -s -A "PASTE-FULL-OAI-SEARCHBOT-STRING" https://www.example.com/pricing | head -c 800

5. Repeat for /robots.txt itself. Anthropic warns that IP blocking stops it reading your robots.txt file. Our robots.txt reachability check explains why that matters.

6. Check your CDN or WAF dashboard. Look at firewall events filtered by the crawler names, and at any AI crawler or bot category settings for your zone.

Compare each crawler's result with your browser baseline.

Prioritise by consequence, as of October 2026. A block on OAI-SearchBot, PerplexityBot or Claude-SearchBot affects search and answer eligibility. A block on GPTBot or ClaudeBot affects training use only. OpenAI says each of its settings is independent of the others, so blocking GPTBot does not remove you from ChatGPT search. OpenAI's crawlers and firewalls walks through each one.

The tempting fix is to allowlist the crawler's IP ranges and move on. OpenAI publishes those ranges, for example at openai.com/searchbot.json, and also warns against relying on IP-only allowlisting.

The reasons are practical. Ranges change. OpenAI's published files list IPv4 prefixes only, with no IPv6. And an allowlist written for one bot does nothing for the others.

The safer pattern:

• Turn on your CDN's verified-bot handling, which checks bots by signature, published IP list with a stable user agent, or reverse DNS.
• Allow the specific search crawlers by user agent combined with that verification.
• Keep challenges for unverified traffic claiming to be a bot.
• Re-test after every change, and after any vendor announcement about default settings.

Spoofing a user agent from your laptop is not the same as being the crawler.

• False alarms. A WAF that verifies bots by IP or reverse DNS will block your spoofed request, while letting the real crawler through. A 403 here means "check the rules", not "confirmed broken".
• False passes. A rule that blocks datacenter IP ranges will let your home connection through and still stop the real crawler.
• One page at a time. Rules often differ by path, so a blog post can pass while pricing fails.
• Answer-time fetchers behave differently. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Your firewall can still block both.

This is also why crawler checkers can give different results. Passing this test buys eligibility only. It does not guarantee retrieval or citation.

The by-hand test takes ten minutes per page. The free AI Crawler Checker on 99Visibility requests a page as each AI crawler and shows which ones your firewall turns away, so you can read every bot's result in one view.

Use it as the first pass. When a crawler shows as blocked, open your WAF event log for that request, find the rule and apply the verified-bot fix above.

Is my CDN blocking AI crawlers even if I never changed anything? It can be. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers stay allowed by default. Other CDNs and plugins ship their own bot defaults and update them. The only way to know is to test the page and check the bot settings for your zone.

Should I unblock GPTBot to appear in ChatGPT? Not for that reason. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, and says each setting is independent. Whether you allow GPTBot is a training decision. For ChatGPT search, what matters is that robots.txt allows OAI-SearchBot and your firewall or CDN does not refuse it.

What does a 403 to an AI crawler actually mean? A 403 means your server or something in front of it refused the request. For a real search crawler, that page is unreachable, so it cannot enter that engine's index. From a by-hand test, it means a rule matched your spoofed request. Confirm it in your WAF event log, because some firewalls block fake bots on purpose while letting verified ones through.

Can I block AI training bots but keep AI search access? Yes. Vendors separate training crawlers from search crawlers. OpenAI splits GPTBot from OAI-SearchBot, and Anthropic splits ClaudeBot from Claude-SearchBot. You can disallow the training bots in robots.txt and in your CDN's training category while allowing the search bots. Then re-run the test for each search crawler to confirm the firewall honours the split.

How often should I re-run the firewall test? Re-run it after any change to your CDN, WAF, security plugin or hosting, and after your CDN announces new bot defaults. A monthly check on your most important pages is a sensible baseline. Blocks are silent: nothing in a standard analytics dashboard tells you a crawler started getting 403s.