Verifying Real AI Crawlers vs Spoofed User Agents

Every HTTP request carries a user-agent header. The client writes it, so the client can write anything. A scraper can call itself GPTBot or PerplexityBot in one line of configuration.

That creates two opposite problems:

• Trusting fakes. If your firewall waves through anything that says OAI-SearchBot, scrapers learn to say OAI-SearchBot.
• Blocking the real thing. If you block by user agent or by a rough IP rule, you can turn away the genuine crawler. OpenAI warns that bot-protection systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."

The second problem is the costly one for visibility. OpenAI's help centre describes ChatGPT search eligibility as a conjunction: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. A robots.txt allow means nothing if the network layer returns a 403. Our page on OpenAI crawlers, robots.txt and firewalls walks through that two-part gate.

So the goal is not to block bots. The goal is to know which requests are really from the operator they claim, then decide on that basis.

Cloudflare's verified-bot criteria are a useful map of the options. A bot qualifies through "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS," plus robots.txt compliance. Each one ties the request to infrastructure the operator controls.

Published IP list plus stable user agent

The operator publishes the address ranges its crawler uses. You check that the request's source IP falls inside the range for the bot named in the user agent. OpenAI publishes separate JSON files per crawler, including searchbot.json for OAI-SearchBot, gptbot.json, chatgpt-user.json and adsbot.json. When the evidence base checked them in August 2026, all four listed IPv4 prefixes only, with no IPv6 entries. That matters if your origin is IPv6-only, which we cover in whether ChatGPT's crawlers can reach an IPv6-only site.

Reverse DNS with forward confirmation

You look up the hostname for the source IP, check it belongs to the operator's documented domain, then look that hostname up again and confirm it resolves to the same IP. The round trip matters: anyone can set a misleading reverse record on their own IP, but they cannot make the operator's domain point back to it. The full method is in our reverse DNS verification entry. Use it only where the operator documents a hostname pattern. Check each operator's crawler page rather than assuming.

Verified-bot programmes and signatures

CDNs and bot-management vendors maintain lists of verified bots and do the checking for you. Signature schemes such as Web Bot Auth go further: the request carries a cryptographic signature you can validate. If your CDN offers a verified-bot category, it is usually the most reliable switch you can flip.

Run this against a real log line that claims to be an AI crawler.

1. Pull the claim. Record the user-agent string, source IP, timestamp, requested URL and status code from your origin or CDN log.
2. Identify the operator and bot. Map the user agent to the operator's documented bot name. For OpenAI, the bots page separates GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (user-initiated actions). Each has its own IP file.
3. Fetch the current IP list for that exact bot. Download it fresh. Do not reuse a copy from months ago.
4. Check the IP against the list. Confirm the source IP sits inside one of the listed prefixes for that bot, not a different bot from the same operator.
5. Run reverse DNS where documented. Resolve the IP to a hostname, confirm the domain matches the operator's documented pattern, then resolve the hostname forward and confirm it returns the same IP.
6. Check your CDN's verdict. See whether your bot-management layer classified the request as a verified bot, and in which category.
7. Decide. If the user agent and at least one operator-controlled signal agree, treat the hit as real. If they disagree, treat it as unverified traffic.
8. Check what the real crawler received. For verified hits, look at the status code. A 200 is a pass. A 403, a 429, or an HTML challenge page means your stack is blocking a genuine crawler.

If you'd rather start without touching logs, our free way to verify a real AI crawler hit covers a lighter version of the same checks.

OpenAI publishes its IP ranges and, in the same breath, advises against relying on IP allowlisting alone. That looks contradictory. It isn't.

IP lists are a verification input, not a complete access policy. Ranges change, and a hardcoded allowlist goes stale. When OpenAI adds a prefix your list lacks, the real crawler hits a wall and nothing alerts you. A list that only covers IPv4 also says nothing about other paths into your origin.

The practical reading, as of October 2026: validate on the user agent plus a verified-bot programme or published list, fetched fresh. Don't build a firewall rule that says "only these IPs, forever." Our answer on whether to allowlist OpenAI's crawler IPs goes into the trade-offs, and the IP allowlisting glossary entry covers the general caveats.

One more reason to avoid blunt IP blocks: Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." If a crawler can't read robots.txt, it can't read your rules either.

Work through this list once, then keep it for every repeat run.

• Each AI crawler you care about is mapped to its exact user-agent token and its operator's documentation page.
• Training bots and search bots are listed separately (for OpenAI: GPTBot vs OAI-SearchBot), because blocking one does not block the other.
• IP lists are fetched automatically from the operator's published URLs, not pasted into a config file.
• Each hit is matched to the IP list for the bot it names, not just any list from that operator.
• Reverse DNS with forward confirmation is used only where the operator documents a hostname pattern.
• Your CDN's verified-bot setting is on, and you know which categories (search, agent, training) it allows.
• /robots.txt itself returns 200 to each verified crawler.
• A sample content URL returns 200, not 403, 429 or a challenge page, to each verified crawler.
• User-triggered fetchers are handled knowingly: OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules."
• Requests that claim a known crawler name but fail every operator-controlled signal are logged as unverified, not as the crawler.

The "real bot blocked" row is the one that quietly removes you from AI answers. From 2026-09-15, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can land in that row without anyone editing a setting. If unfamiliar names turn up in your logs, our answer on blocking AI crawlers you don't recognize covers how to decide.

To see what each crawler gets back from your site right now, the AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

Verification is not a one-off. The lists, your stack and vendor defaults all move.

• Continuously: fetch operator IP lists on a schedule (daily is cheap) and let your bot rules read the fresh copy. OpenAI's files carry a creation timestamp, and chatgpt-user.json was regenerated in August 2026.
• After any change: rerun the reachability check after a CDN migration, a new WAF rule, a bot-management setting change or a hosting move.
• After vendor announcements: recheck when a CDN changes defaults, as Cloudflare did on 2026-09-15.
• Monthly otherwise: sample a week of logs, verify the AI crawler hits, and confirm each real crawler got 200s.

What this controls: whether real crawlers reach you and fake ones get no special treatment. What it doesn't control: whether any engine retrieves or cites your page. Access is the first gate, not a guarantee.

Can I just allowlist OpenAI's published IP ranges and stop there? Not safely. OpenAI publishes the ranges and also warns against IP-only allowlisting. Ranges change, and a static allowlist goes stale without telling you. Use the lists as a fresh verification input, paired with the user agent and, where available, your CDN's verified-bot programme. Keep the rule pointed at the live list rather than a pasted copy.

Do OpenAI's crawlers use IPv6? When the evidence base checked OpenAI's four IP files in August 2026, all listed IPv4 prefixes only, with zero IPv6 entries. If your origin or bot-facing edge only resolves over IPv6, OpenAI's crawlers may not reach it regardless of robots.txt. Recheck the current files before acting, because operators can update them at any time.

If a request claims to be GPTBot but fails verification, should I block it? Treat it as ordinary unverified traffic and apply whatever policy you use for unknown clients. Blocking a spoofed request does not affect the real GPTBot, which comes from OpenAI's published ranges. What you should never do is give a failed-verification request the access you reserve for the genuine crawler.

Do user-triggered AI fetchers follow robots.txt? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiates those fetches. Perplexity says Perplexity-User generally ignores robots.txt rules. Identity verification still works on these requests, but if you want to control them, do it at the network or CDN layer, not in robots.txt.

Will verifying crawlers help my brand get cited in AI answers? Verification doesn't earn citations. It makes sure the real crawlers aren't silently blocked and that impostors aren't trusted. Being reachable is a precondition for search crawlers like OAI-SearchBot to include you, but retrieval and citation are decided by each engine per query, and no site setting controls that.