Reviewing CDN Bot Settings for AI Crawlers

Your content delivery network sits between AI crawlers and your server. If it refuses a crawler, your robots.txt never matters. The crawler gets a 403 or a challenge page, and nothing in your analytics tells you.

OpenAI states this plainly in its help article on ChatGPT search. To be eligible, a site must allow OAI-SearchBot to crawl it and confirm that its host or CDN allows traffic from OpenAI's published search bot IP addresses. Both conditions apply. Passing one is not enough.

The risk is that CDN providers change their defaults. On September 15, 2026, Cloudflare began blocking Training and Agent crawlers by default on pages that display ads. The change covers all new domains, new sites added to existing accounts, and all existing free-tier customers. Search crawlers stay allowed by default.

That means a site can lose some AI access without its owner touching a setting. If you want the broader picture, read whether AI crawler access can break without you changing anything. The next move is a review of what your CDN actually does today.

Most bot management dashboards now group AI bots by purpose. The labels vary by provider, but the three common buckets are search, agent and training. Each one maps to different crawlers with different consequences.

The consequences come from the operators' own documentation. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI also says GPTBot covers training only, and that each setting is independent of the others. Blocking GPTBot does not remove you from ChatGPT search.

Two cautions apply. First, your provider decides which bucket each crawler lands in. Check the label for every crawler you care about rather than assuming. Second, the evidence shows no measured effect, in either direction, from allowing or blocking training crawlers. Treat training as a policy choice, not a visibility lever.

Set aside 30 minutes. You need admin access to your CDN dashboard and your robots.txt file open in another tab.

1. Find every zone or site on the account. Defaults can differ between an old site and a newly added one. List each domain and subdomain that serves public content.
2. Open the bot-management or AI crawler section for each zone. Note the plan tier. Free-tier accounts were in scope for Cloudflare's September 2026 change.
3. Record the setting for each category. Write down search, agent and training as allowed, blocked or challenged. A challenge (a JavaScript check or CAPTCHA) counts as a block for an automated crawler.
4. Check page-level scope. Some defaults apply only to pages that display ads. Note which templates on your site carry ads.
5. Look for custom rules that override categories. Firewall rules, rate limits, country blocks and IP allowlists can block a crawler the category allows. Search the rules list for user-agent and IP conditions.
6. Compare against robots.txt. For each AI crawler, your CDN and robots.txt should agree. The difference between the two gates is where most silent failures live.
7. Decide what you want, per category. Most businesses that want to appear in AI answers allow search. Agent and training are separate decisions.
8. Change the settings and write down the date. A change log lets you connect a later drop in AI citations to a specific edit.
9. Test with real requests. Request a sample of URLs as each crawler and read the response. The firewall testing playbook walks through this, and the AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Do not rely on IP allowlisting alone. OpenAI publishes IP ranges for OAI-SearchBot, but the evidence base notes OpenAI also warns against IP-only allowlisting. Prefer your provider's verified-bot programme plus user-agent checks.

Use this list each time you run the review. Every line should end with a yes or a written reason.

• Every public zone and subdomain is listed, including recently added ones.
• The plan tier for each zone is recorded.
• The search category is allowed on every zone where you want AI search visibility.
• Agent and training categories are set by a deliberate decision, with the reason noted.
• No challenge or CAPTCHA is served to verified AI crawlers.
• No custom firewall rule, rate limit or country block catches AI crawler traffic by accident.
• robots.txt itself returns 200 to AI crawlers. Anthropic warns that IP blocking impedes its ability to read your robots.txt file.
• robots.txt rules and CDN categories agree for each named crawler.
• Test requests as OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer return 200 on sample pages.
• Ad-displaying templates were tested separately from pages without ads.
• The review date and any changes are logged.

Good results

• Search crawlers get a 200 response with your real HTML on every sample page.
• robots.txt returns 200 to every crawler you tested.
• The dashboard setting, the robots.txt rule and the test result all say the same thing.
• Any block on training or agent crawlers is one you chose and wrote down.

Bad results

• A 403 to a search crawler. For OAI-SearchBot, that is a complete and silent loss of ChatGPT search eligibility. Start with the 403 troubleshooting playbook.
• A 200 that is actually a challenge page. Read the body, not just the status code. An HTML page with a JavaScript check or CAPTCHA is a block.
• A 429. Rate limiting can turn away crawlers that arrive in bursts.
• Pages with ads behave differently. That pattern points to a page-scoped default like Cloudflare's September 2026 change.
• Settings and tests disagree. A custom rule is overriding the category. Find it before changing anything else.
• A blanket AI block. The most common real-world error is one switch that blocks training and search together, silently removing ChatGPT search eligibility.

Access can change from three directions: your team, your provider and the crawler operators. Schedule the review around all three.

• Monthly, as a standing check. It takes minutes once your checklist exists.
• After any account change: a plan downgrade, a new zone, a migration or a new security vendor.
• After any provider announcement about bot defaults or AI crawler categories. Cloudflare announced its change ahead of the date, so watch your provider's changelog.
• After any firewall or WAF rule change, even one aimed at unrelated traffic.
• When an engine stops citing you. If you suspect Cloudflare specifically, see whether Cloudflare's AI crawler settings could be hiding your site.

The monthly cadence is our recommendation, not a published standard. Pick a rhythm you will actually keep.

A clean CDN review removes a block. It does not create a citation. Allowing a search crawler buys eligibility for that engine's index, not retrieval and not ranking. Whether an engine pulls your page into an answer for a given question is decided at answer time by the engine.

Some traffic also sits outside robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user starts the request. Perplexity says Perplexity-User generally ignores robots.txt rules. For those agents, the CDN setting is the control that actually takes effect.

Google works differently. Googlebot governs AI Overviews and AI Mode, and there is no separate robots.txt token for them. Blocking Googlebot at the CDN affects all of Google Search at once.

The practical line: keep the gates open for the crawlers you want, confirm it with tests, and re-check on schedule. What the models say after that is not in your control.