When Your CDN or Firewall Blocks AI Crawlers

Every AI crawler passes two separate gates before your content can reach an engine's index. The first is robots.txt, a text file that tells each bot what it may fetch. The second is the network layer: your content delivery network, your web application firewall, your bot management rules and any IP allowlists.

The two gates do not talk to each other. robots.txt can say Allow while the firewall returns 403 Forbidden to the same bot. When that happens, the crawler never sees your page, and nothing in your CMS or analytics tells you.

OpenAI spells this out for ChatGPT search. Its help article says that to make a site eligible, you allow OAI-SearchBot to crawl and confirm that the host or CDN allows traffic from OpenAI's published search bot IP addresses. The word is "and". robots.txt alone is not enough behind a blocking firewall.

If you want the side-by-side version, read robots.txt vs firewall: the two gates for AI crawlers. The rest of this page is about the second gate.

Firewalls and bot managers are built to stop automated traffic. AI crawlers are automated traffic. Most blocks are side effects of rules written for scrapers, credential stuffers or DDoS protection.

The common mechanisms are:

• Bot score or challenge rules. The firewall serves a JavaScript challenge or CAPTCHA to anything that looks automated. OpenAI's own guidance notes that because its crawlers are automated systems, these checks may block access, and that such systems can mistakenly block legitimate crawlers, often returning 403 errors.
• A one-click "block AI bots" setting. Many CDNs and security plugins offer a switch that blocks a whole list of AI user agents. That list usually mixes training crawlers with search crawlers. If you run WordPress, see whether security plugins block AI crawlers.
• Rate limits. A crawler that fetches many URLs quickly hits a threshold and gets 429 Too Many Requests.
• Country or ASN blocks. Rules that block datacenter IP ranges or whole regions catch crawler traffic that comes from cloud providers.
• IP allowlists that are out of date. A site that only admits known IPs can miss new ranges. OpenAI publishes its IP ranges as JSON files, and as of its August 2026 files they listed IPv4 ranges only. An origin that only accepts IPv6 cannot be reached from those ranges at all.

One block has a knock-on effect. Anthropic warns that IP blocking impedes its ability to read your robots.txt file. If the crawler cannot fetch /robots.txt, your per-bot rules never get read.

On 15 September 2026, Cloudflare changed its default bot settings. Its Training and Agent crawler categories are now blocked by default on pages that display ads. Search crawlers remain allowed by default. The change applies to new domains, to new sites added by existing customers, and to all existing free-tier customers.

This matters for one reason: a site can lose some AI crawler access without its owner editing a single setting. If you are on a free Cloudflare plan and run ads, your configuration may have changed under you.

What it means in practice depends on which category a bot falls into:

• Blocking a training crawler such as GPTBot only affects whether your content is used to train future models. OpenAI states that each of its crawler settings is independent. Blocking GPTBot does not remove you from ChatGPT search.
• Search crawlers, such as OAI-SearchBot, stay allowed under the new default. Those are the ones that decide whether you can appear in AI search answers.
• Agent fetchers act on behalf of a user in real time. Cloudflare's default now blocks that category on ad-displaying pages.

The next move is simple. Open your Cloudflare dashboard, find the AI crawler or bot settings for each zone, and confirm that the categories match what you actually want. Nobody outside your account can see this setting.

Access is the most controllable part of AI visibility. It is also the part with the clearest evidence, because the engine operators document the mechanism themselves.

In your control:

• Which crawlers your robots.txt allows, per bot.
• Whether your CDN, firewall and security plugins let those same crawlers through.
• Whether /robots.txt itself is reachable to every crawler.
• Whether your allowlists use user agent plus verified-bot programmes rather than IP lists alone. OpenAI publishes IP ranges and also warns against relying on IP-only allowlisting.

Not in your control:

• Whether an engine retrieves your page for a given question. Retrieval is per query and at the engine's discretion.
• Whether a retrieved page gets cited, and what the answer says.
• How user-triggered fetchers behave. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules. Your firewall still sees them.
• What the model already knows from training, which no access change reaches this quarter.

Be plain with your team about the payoff. Fixing a firewall block removes a reason you are absent. It does not guarantee a single citation. Nobody has published how often robots.txt and firewall settings disagree across sites, so treat this as a thing to check, not a known epidemic.

Run the test in this order. Each step catches something the previous one misses.

1. Read your robots.txt per bot. List the retrieval crawlers you care about: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot and bingbot. Note which are allowed. Our per-bot robots.txt guide covers what each one controls.
2. Request pages as each crawler. Send a request to your homepage, your pricing page and one deep content page using each crawler's documented user-agent string. Request /robots.txt the same way.
3. Repeat from two networks. Send one set of requests from a datacenter connection and one from a residential connection. Many firewall rules treat the two differently.
4. Classify every response using the table below.
5. Check your logs for the real bots. Filter your CDN or origin logs to verified crawler traffic and look at the status codes those bots actually received.
6. Check the CDN dashboard. Review bot categories, managed rules and any AI crawler settings, especially after 15 September 2026 on Cloudflare.

Steps 2 and 3 are a fast screen, not a verdict. A spoofed user agent from your laptop is not the real crawler. A good bot manager may correctly block your fake request while admitting the real one, or admit yours while blocking real crawler IPs. That is why AI crawler checkers can give different results, and why step 5 settles it. The full procedure is in our playbook on testing whether your firewall lets AI crawlers in.

For a quick first pass, the free AI Crawler Checker on 99Visibility requests a page as each AI crawler and shows which ones the firewall turns away.

The goal is to admit the specific crawlers you chose, not to switch off bot protection.

• Allow by verified identity. Cloudflare describes verified bots as those that prove identity through a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. Use your CDN's verified-bot feature rather than a raw user-agent match, which anyone can fake.
• Split training from search. Decide separately. You can block GPTBot for training and still allow OAI-SearchBot for ChatGPT search.
• Exempt /robots.txt from challenges. Every crawler needs to read it before anything else works.
• Retest after every change. Rerun the response checks and read the logs again a few days later.
• Write it down. Note which rule you changed and why. The next person to tighten security should know that rule matters.

• Assuming robots.txt is the whole story. It is one gate of two. A clean robots.txt behind a blocking firewall is a silent failure.
• Flipping a blanket "block AI" switch. Most of these lists mix training and search crawlers. Blocking the search crawler removes eligibility for that engine's search answers.
• Believing GPTBot controls ChatGPT search. It governs training only. OAI-SearchBot is the search crawler.
• Testing only the homepage. Firewall rules often differ by path. Test pricing, product and content pages too.
• Trusting one spoofed test. Confirm with logs of verified bot traffic before you conclude anything.
• Relying on IP lists alone. Ranges change, and OpenAI itself advises against IP-only allowlisting.
• Forgetting the plan changed. A CDN default can change your access without any edit on your side. Recheck after vendor announcements.