Web application firewall (WAF): definition and AI crawler impact

A web application firewall sits between visitors and your web server. It reads each HTTP request and applies rules to decide whether to pass it, challenge it or reject it.

WAFs exist to stop attacks such as injection attempts, credential stuffing and abusive scraping. Many run inside a content delivery network, a hosting control panel or a security plugin, so site owners often have one without having set it up themselves.

This entry is a short definition. For the full diagnosis and fixes, read the guide on when your CDN or firewall blocks AI crawlers.

A WAF typically weighs several signals at once:

• User-agent string: the name the client claims, such as OAI-SearchBot or PerplexityBot.
• IP address and reputation: datacenter ranges often score as higher risk than home connections.
• Behaviour: request rate, missing browser features, or failing a JavaScript challenge.
• Verified-bot status: Cloudflare, for example, treats some crawlers as verified bots based on signals such as published IP lists or reverse DNS.

The outcome is usually one of three: a normal 200 response, a challenge page (a CAPTCHA or JavaScript check), or an error such as 403 Forbidden or 429 Too Many Requests. An automated crawler can't solve a challenge, so for a crawler a challenge works like a block. Much of this overlaps with bot management, which is the broader category of tools that classify automated traffic.

AI search engines need to fetch your pages before they can use them. robots.txt is one gate. The WAF is a second, independent gate.

OpenAI's help page on ChatGPT search states the conjunction plainly: allow OAI-SearchBot to crawl the site, and confirm your host or CDN allows traffic from OpenAI's published search bot IP addresses. robots.txt alone is not enough if the firewall says no. Anthropic notes that blocking its IP addresses also stops it reading your robots.txt file, so your stated rules never reach it.

Defaults can change without you editing anything. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains and existing free-tier customers, with Search allowed by default. As of October 2026, that is worth confirming on any Cloudflare-fronted site.

How often a WAF blocks a crawler that robots.txt allows has not been measured by anyone. So the next move is a test, not an assumption: request a page as each AI crawler and record the status code. The robots.txt vs firewall comparison explains the two gates side by side, and the AI Crawler Checker at 99Visibility requests a page as each AI crawler and shows which ones the firewall turns away.

• "If robots.txt allows a bot, it gets in." No. robots.txt is a request to well-behaved crawlers. The WAF enforces access and can override it with a 403 or a challenge.
• "Blocking all AI bots only stops training." A blanket rule can also catch retrieval bots such as OAI-SearchBot, which OpenAI uses for ChatGPT search. OpenAI documents its bots as separate settings, so block each training crawler by name instead.
• "Allowlisting IPs solves it." OpenAI publishes IP ranges but also warns against relying on IP-only allowlisting. Validate on user-agent plus verified-bot programmes.
• "Getting past the WAF gets you cited." Reachability is eligibility only. What the engine retrieves and cites is outside your control.

• Web crawler: the automated client the WAF is judging.
• Content delivery network: often where the WAF runs.
• Bot management: the classification layer that labels crawlers as good, bad or unknown.
• Training crawler: bots such as GPTBot that collect content for model training, not search.

Is a WAF the same as a CDN? No. A CDN caches and serves your content from servers close to visitors. A WAF filters requests for threats. Many CDNs include a WAF and bot management in the same product, which is why a CDN setting can block an AI crawler. Check the security and bot rules in your CDN dashboard, not just its caching settings.

What does a WAF block look like to an AI crawler? Usually a 403 Forbidden, a 429 rate-limit response, or an HTML challenge page asking for a CAPTCHA or JavaScript check. The crawler cannot complete a challenge, so it leaves with nothing. Your analytics may show no error at all, because the request never reached your server. That silence is why a direct test is the only reliable way to know.

Can a WAF stop AI crawlers from reading robots.txt? Yes. If the firewall blocks a crawler's requests, it blocks the request for /robots.txt too. Anthropic notes that IP blocking impedes its ability to read your robots.txt file. The crawler then cannot see your allow rules, so test whether robots.txt itself returns 200 to each AI crawler you care about.

Should I turn my WAF off to let AI crawlers in? No. The WAF protects your site from real attacks. The fix is a targeted exception: allow the specific retrieval crawlers you want, using verified-bot settings and user-agent rules, while keeping protection on for everything else. Test again after any rule change, because a later update or new default can undo it.

Does WordPress have a WAF? WordPress core does not, but many security plugins and managed hosts add firewall and bot-blocking rules. These can block AI crawlers as a side effect. If you run WordPress, review the plugin's bot and rate-limit settings and test each crawler's access, since the plugin's defaults decide what gets through.