Content delivery network (CDN): definition and AI crawler impact

A content delivery network is a layer of servers between your website and everyone who requests it. It caches pages, images and scripts at many locations, so visitors get a fast response from a nearby server. It also inspects incoming traffic and can block, challenge or rate-limit requests it thinks are harmful.

That second job is the one that matters for AI search. A CDN is not only a speed tool. It is a gatekeeper, and AI crawlers have to pass it like any other visitor. For the full failure mode and how to fix it, see the guide on when your CDN or firewall blocks AI crawlers.

When someone types your domain, DNS points them at the CDN rather than straight at your server. The CDN then does three things:

• Serves a cached copy if it has one, or fetches a fresh copy from your origin server.
• Filters the request using rules such as a web application firewall, IP reputation and rate limits.
• Classifies automated traffic through bot management, sorting requests into categories like verified search bots, AI training crawlers and unknown bots.

Each category gets a policy: allow, challenge or block. Many site owners never open these settings. They inherit whatever the provider ships as the default.

AI crawlers fetch pages so engines like ChatGPT, Perplexity and Claude can index and quote them. If the CDN blocks the crawler, the engine never sees the page, whatever your robots.txt says.

OpenAI's help center spells this out. To make a site eligible for ChatGPT search, you allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Both conditions apply. Anthropic adds that IP blocking can stop it reading your robots.txt file at all.

Defaults can also change under you. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on pages that display ads. That applies to new domains, new sites on existing accounts and all existing free-tier customers, while Search crawlers stay allowed by default. A site can lose some AI crawler access without anyone on the team editing anything, and nothing in robots.txt shows it. Read more on whether Cloudflare's AI crawler settings could be hiding your site.

The next move is simple: request your pages as each AI crawler and compare the result with your robots.txt. The AI Crawler Checker does this and shows which crawlers the firewall turns away. Then work through reviewing CDN bot settings for AI crawlers.

• "robots.txt controls who can crawl me." It states your rules. The CDN enforces its own. They are two separate gates, and a crawler has to pass both.
• "Blocking AI bots is all or nothing." Crawlers have different jobs. OpenAI says GPTBot governs training use and OAI-SearchBot governs ChatGPT search, and each setting is independent. A CDN rule that blocks a training crawler need not touch search crawlers, but a blanket AI block will.
• "Allowlisting IP ranges is enough." OpenAI publishes IP ranges but warns against relying on IP-only allowlisting. Verify by user agent and the provider's verified-bot programme too.
• "Passing the CDN gets me cited." Reachability only makes a page eligible. What the engine retrieves and cites is outside your control.

• Web application firewall (WAF): the rule engine, often inside the CDN, that blocks requests.
• Bot management: the system that classifies crawlers and applies per-category policies.
• Training crawler: a bot that collects content for model training, distinct from search crawlers.
• robots.txt: the file that states your crawl rules, compared with firewalls in robots.txt vs firewall.