Do WordPress security plugins block AI crawlers?

robots.txt is a text file that tells polite crawlers what they may fetch. It does not enforce anything. Your security plugin, your host's firewall and your CDN sit in front of WordPress and decide whether a request gets a page at all.

Those layers judge requests by user agent, IP address, request rate, country or behaviour. An AI crawler can match a rule written for scrapers. When that happens, the crawler gets a 403 Forbidden, a rate-limit 429, or a challenge page instead of your content. Your robots.txt still says "allow", and no WordPress dashboard tells you anything went wrong.

OpenAI spells out the two-part requirement. Its help page on ChatGPT search says to allow OAI-SearchBot to crawl the site and to confirm that your host or CDN allows traffic from OpenAI's published searchbot IP addresses. The word is "and". One gate open is not enough. The full comparison is in robots.txt vs firewall: the two gates for AI crawlers.

What we cannot tell you is how common this is. Nobody has published a rate for WordPress plugins, CDNs or hosts. The mechanism is documented by the crawler operators. The frequency is unmeasured.

Each AI company runs several bots, and each one has a different job. Blocking the wrong one at the firewall has a different cost from blocking it in robots.txt.

OpenAI states that each setting is independent of the others. So a firewall rule that blocks anything with "GPT" in the user agent can catch more than you meant. See OpenAI's crawlers, robots.txt and firewalls for the detail, and whether to block GPTBot if you want ChatGPT search for the training versus search choice.

Google is different. Googlebot handles AI Overviews and AI Mode, and there is no separate robots.txt token for them. If your firewall blocks Googlebot, you have a bigger problem than AI search.

Run the test on a page that matters, such as pricing or a main service page, not just the homepage.

1. Copy the exact user agent string for each bot from its operator's crawler page. Firewalls often match on parts of that string.
2. Request the page with each user agent and record the status code.
3. Request /robots.txt the same way. Anthropic warns that IP blocking can stop it reading your robots.txt at all.
4. Look at the body, not only the code. A 200 that returns a challenge or CAPTCHA page is still a block.

A minimal command looks like this, with the full string pasted in:

curl -s -o /dev/null -w "%{http_code}\n" -A "<full OAI-SearchBot user agent>" https://example.com/pricing/

Repeat it for OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot and PerplexityBot. A 200 for a normal browser and a 403 for a bot is the pattern to look for. The step-by-step version is in testing whether your firewall lets AI crawlers in. If you would rather not use a terminal, the AI Crawler Checker from 99Visibility requests a page as each AI crawler and shows which ones get turned away.

A user agent test from your laptop has limits. Know them before you trust the result.

• Spoofed requests are not real bots. Some firewalls verify crawlers by IP address. Your test request carries the right name but the wrong IP, so it may be blocked even though the real crawler gets through, or the reverse.
• Logs are the ground truth. Filter your host, plugin or CDN logs for each AI user agent and check the status codes the real crawlers received.
• IP allowlists go stale. OpenAI publishes its search crawler ranges at a JSON endpoint and also warns against relying on IP allowlisting alone. Validate on user agent plus your provider's verified-bot features.
• Rate limits look random. A crawler can pass on one request and get a 429 on the next. Test more than once.
• Settings can change without you. Per the 99Visibility evidence base, Cloudflare announced it would block Training and Agent crawler categories by default on ad-displaying pages from 15 September 2026 for new domains and existing free-tier customers. A CDN in front of WordPress can change your access while you edit nothing.

For CDN-specific causes, read when your CDN or firewall blocks AI crawlers.

Fix the layer that returned the block, not the one you happen to be logged into.

• Security plugin: look for bot blocking, "fake crawler" detection, rate limiting and country rules. Add an exception for the specific AI crawlers you want, by verified user agent.
• Host firewall: some managed WordPress hosts filter bots before WordPress loads. Ask support which rule matched and request an exception.
• CDN: check its AI or bot categories. Search crawlers and training crawlers are often listed separately, so you can allow one and block the other.
• robots.txt: make sure it is reachable and says what you intend per bot. The per-bot robots.txt guide covers the syntax.

Then run the test again. Getting through both gates makes your pages eligible to be fetched. It does not make any engine cite you. What happens after access depends on retrieval and the model, which you do not control.

Will a 403 to GPTBot remove me from ChatGPT search? Not by itself. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the search crawler, and says each setting is independent. A 403 to GPTBot affects training collection. A 403 to OAI-SearchBot is the one that matters for ChatGPT search eligibility. Check your firewall rules for broad patterns, because a rule written for one bot often catches the other.

Is it safe to allowlist AI crawlers by IP address? It helps, but do not rely on it alone. OpenAI publishes IP ranges for its crawlers and also warns against IP-only allowlisting. Ranges change, and a stale list blocks the real bot. Combine user agent rules with your firewall or CDN's verified-bot features, and recheck your logs after any change to the list.

Can I block user-triggered AI fetches with robots.txt? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiated the request. Perplexity says Perplexity-User generally ignores robots.txt. If you want to stop those fetches, the firewall is the only layer that enforces it. Decide deliberately, since these fetches happen when a user asks about your page.

Does WordPress's built-in robots.txt block AI bots? A default WordPress install generates a virtual robots.txt that does not target AI crawlers. SEO and security plugins can change it, and some add AI bot rules. Open yoursite.com/robots.txt to see what is actually served, then test whether AI crawlers can fetch it. A [reachability check for robots.txt](/free/free-robots-txt-reachability-check) shows whether the file loads for them.