Why do AI crawler checkers give different results for my site?

A crawler checker cannot be the crawler. It fetches your page from its own server and sets the user agent string to something like OAI-SearchBot. Your firewall then makes a decision about that one request.

The real OAI-SearchBot arrives from OpenAI's own IP ranges, at its own pace, with its own history on your network. Your firewall makes a separate decision about it. Those two decisions can differ.

So when two tools disagree, both may be reporting accurately. One tool's network got through. The other's got challenged or blocked. Neither result tells you for certain what the real bot saw.

This matters because the failure is silent. OpenAI's help page notes that bot protection systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." Your robots.txt can say allow while your edge says no, and no dashboard flags it. Our guide to CDN and firewall blocking of AI crawlers covers the mechanism in more depth.

Bot management tools judge requests on several signals at once. Three explain most checker disagreements.

IP origin

A request from a cloud datacenter looks different from one on a home broadband line. Many firewall rules score datacenter traffic as higher risk. A checker running in a datacenter may get challenged, while a test from your laptop passes. The reverse also happens when a rule allowlists known crawler ranges but the checker's range is not on it.

User agent

Some rules match the user agent string directly, for example a blanket block on anything containing GPTBot or ClaudeBot. A checker that sends the right string will trip that rule. A checker that sends a generic browser string will not.

Request rate

A checker sends a handful of requests. A real crawler may send many over a short window. Rate limits that never fire during a test can return 429 to the real bot. JavaScript challenges and CAPTCHAs behave the same way: OpenAI notes that because its crawlers are automated, "these checks may block access." See whether CAPTCHAs and bot challenges block AI crawlers for that case.

This is the same pattern behind why AI visibility tools disagree more broadly: different inputs, different outputs.

Anyone can type OAI-SearchBot into a request header. That is why serious bot management does not trust the user agent alone.

Cloudflare's documentation describes verified bots as those identified by a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. A checker that spoofs the string has none of these. A well-configured firewall may block the spoof and let the genuine crawler through. A poorly configured one may do the opposite.

OpenAI publishes its search crawler's IP ranges at openai.com/searchbot.json and states that eligibility requires two things: allow OAI-SearchBot in robots.txt, and confirm your host or CDN allows traffic from those published IP addresses. The second half is what a spoofed test cannot reproduce. OpenAI also warns against relying on IP allowlisting alone, so validating on user agent plus verified bot status is the safer configuration.

A few conditions produce results that look random until you know to check them.

• robots.txt itself is blocked. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." A checker may read your page fine while the crawler cannot read the file that grants it permission. Run a robots.txt reachability check separately.
• IPv6-only origins. OpenAI's published crawler IP files list IPv4 prefixes only. If the path a bot takes resolves only over IPv6, it cannot connect, whatever robots.txt says.
• CDN default changes. From 2026-09-15, Cloudflare began blocking Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers stay allowed by default. A test from before that date may no longer reflect your setup.
• Plugins and app-level rules. Security plugins can block on user agent before the CDN is involved. See whether WordPress security plugins block AI crawlers.
• User-triggered fetchers. Perplexity says Perplexity-User "generally ignores robots.txt rules," and OpenAI says robots.txt "may not apply" to ChatGPT-User. These behave differently from index crawlers, so lumping them together confuses results.

Use checkers to find suspects, then confirm with evidence from your own server. Here is the order that works.

1. Run a checker from more than one network. Compare a datacenter test with a residential one. If they differ, your firewall is scoring on IP origin. The free firewall AI bot test is a starting point.
2. Record status codes, not just pass or fail. A 200 with a challenge page in the HTML is a failure. A 403 points at bot mitigation. A 429 points at rate limiting.
3. Pull your origin or CDN logs. Filter to requests claiming to be OAI-SearchBot, PerplexityBot, Claude-SearchBot, bingbot and Meta-WebIndexer. Our guide to seeing which AI bots crawl your site walks through it.
4. Verify the bots. Match source IPs against the operator's published ranges or reverse DNS. Discard spoofed hits.
5. Read the status codes the verified bots received. This is the ground truth. If real OAI-SearchBot requests get 403, fix the rule that blocks them.

Fixing access removes a block. It does not guarantee any engine will retrieve or cite you, since that happens at later stages nobody outside the operator controls. If you want a per-crawler snapshot first, the AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away. Treat its result like any checker: a lead to confirm in your logs.