Bot management: definition and how it affects AI crawlers

Bot management is the layer of rules that sorts incoming requests into humans, known bots and unknown bots, then decides what each one gets. It usually runs on a content delivery network or a web application firewall, in front of your server.

The possible outcomes are simple:

• Allow: the request reaches your page.
• Challenge: the requester must pass a JavaScript check or a CAPTCHA first.
• Block: the request gets an error, often a 403 Forbidden.
• Rate limit: the request gets a 429 after too many hits.

It exists for good reasons: scrapers, credential stuffing and load spikes. The side effect is that legitimate AI crawlers can land in the wrong bucket.

Bot management tools identify a crawler from a mix of signals. Cloudflare's documentation lists the ways a bot can be verified: a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS.

OpenAI publishes the IP ranges for OAI-SearchBot and its other crawlers. It also warns against relying on IP allowlisting alone, so the safer setup validates the user-agent together with your provider's verified-bot programme.

Most providers then group AI crawlers into categories. A useful way to think about them:

Each category has a different consequence when blocked. Blocking a training crawler affects future model training. Blocking a search crawler removes eligibility for that engine's search answers.

OpenAI's help page says eligibility for ChatGPT search needs two things: allow OAI-SearchBot in robots.txt, and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. OpenAI also notes that bot protection systems can mistakenly block legitimate crawlers, often with 403 errors, and that challenge checks may block its automated crawlers.

That is the silent failure. robots.txt says yes, the firewall says no, and no dashboard tells you. Anthropic adds that IP blocking stops it from reading your robots.txt at all.

Defaults also move. Cloudflare announced that from 2026-09-15, Training and Agent crawlers are blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stays allowed by default. A site can lose agent access without its owner changing a setting. Our explainer on whether Cloudflare's settings could be hiding your site covers that case.

The next move is to test, not assume. Request a page as each AI crawler and record the status code and any challenge page. The firewall testing playbook walks through it, and 99Visibility's AI Crawler Checker shows which crawlers your firewall turns away. Passing this gate buys eligibility only. It does not make any engine cite you.

• "robots.txt controls everything." It does not. robots.txt and bot management are separate gates, and a crawler has to pass both. See robots.txt vs firewall.
• "A challenge is softer than a block." For an automated crawler, a CAPTCHA is a block.
• "Blocking all AI bots is one decision." OpenAI states each crawler setting is independent. You can block GPTBot from training and still allow OAI-SearchBot for ChatGPT search.
• "Allowing GPTBot helps ChatGPT find me." GPTBot governs training only. Search uses OAI-SearchBot.
• "Blocking agent fetchers in robots.txt is reliable." Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not apply to ChatGPT-User. Network rules are the real control for those.

• Web application firewall (WAF): the rule engine where many bot policies live.
• Content delivery network (CDN): the edge layer that often runs bot management.
• Training crawler: the category most often blocked on purpose.
• For depth on diagnosing and fixing blocks, read when your CDN or firewall blocks AI crawlers.

Is bot management the same as bot protection? Mostly, yes. Vendors use both names for the same function: classifying automated traffic and deciding whether to allow, challenge or block it. "Bot protection" stresses the defensive side, while "bot management" covers the full policy, including which good bots you let through. For AI visibility the question is the same either way: which AI crawlers actually receive your page.

Should I block AI training crawlers? That is a business choice, not a visibility fix. Blocking a training crawler such as GPTBot signals that your content should not be used for model training. OpenAI states this is independent of OAI-SearchBot, which controls ChatGPT search. Nothing published measures how training exclusion affects what models later know about a brand, so decide based on how you want your content used.

How do I know if my bot management is blocking AI crawlers? Request a real content URL using each AI crawler's documented user-agent and record what comes back. A 200 with your content is a pass. A 403, a 429, or an HTML page containing a JavaScript challenge or CAPTCHA is a fail. Also check that robots.txt itself is reachable, and review your provider's AI crawler categories in its dashboard.

Does bot management affect Google AI Overviews? Google's AI Overviews and AI Mode use Googlebot, and a page must be indexed and eligible to show with a snippet to appear as a supporting link. If your bot rules challenge or block Googlebot, that hurts all of Google Search, not just AI features. Most providers treat Googlebot as a verified bot, but confirm it in your settings.