What is a verified bot?

A verified bot is a crawler whose identity has been checked against something the operator publishes or signs. The check answers one question: is this request really from the company it claims to be from?

Cloudflare's bot documentation lists the accepted proofs as a cryptographic Web Bot Auth signature, a published IP list paired with a stable user-agent, or reverse DNS, together with robots.txt compliance. Bots that pass sit in a verified category that site owners can allow or block as a group.

This entry is a short definition. For the step-by-step method, read the playbook on verifying real AI crawlers vs spoofed user agents.

Every HTTP request carries a user-agent header, such as OAI-SearchBot or ClaudeBot. The client writes that header itself. Anyone can copy it, which is why identity checks look at evidence the impostor cannot fake.

• Published IP ranges. The operator lists the addresses its crawler uses. OpenAI publishes its searchbot ranges at openai.com/searchbot.json. A request claiming to be OAI-SearchBot from an address outside that list is not OAI-SearchBot.
• Reverse DNS. You look up the hostname behind the requesting IP, check that it belongs to the operator's domain, then resolve that hostname forward to confirm it returns the same IP. See reverse DNS verification for crawlers.
• Signatures. The crawler signs each request with a key the operator controls. A valid signature ties the request to the operator directly.

OpenAI warns against allowlisting by IP alone. The safer pattern combines the user-agent with a verified-bot programme run by your CDN or firewall vendor.

AI crawlers reach your pages through two gates. robots.txt says whether a bot may fetch. Your host, CDN or firewall decides whether the request actually gets through.

OpenAI's help centre states the conjunction plainly: to be eligible for ChatGPT search, allow OAI-SearchBot to crawl and confirm your host or CDN allows traffic from OpenAI's published searchbot IPs. Bot-management systems can mistakenly block legitimate crawlers, often with a 403. Anthropic adds that IP blocking can stop it reading your robots.txt at all.

Defaults also shift without you editing anything. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. As of October 2026, that means a site can lose some AI access silently.

The next move: check whether real AI crawlers get a 200. The guide on when your CDN or firewall blocks AI crawlers covers the fixes, and 99Visibility's free AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

"Verified means allowed"

Verification confirms identity. Permission still comes from your robots.txt and firewall rules, set per bot. Read the per-bot robots.txt guide for how each AI crawler's rule works.

"Verified bots always obey robots.txt"

Not every fetcher does. Perplexity says Perplexity-User generally ignores robots.txt rules, and OpenAI says robots.txt may not apply to ChatGPT-User, because users trigger those fetches.

"Letting verified bots in gets you cited"

Access buys eligibility only. Retrieval and citation happen later, under the engine's control, and nobody outside can guarantee them.

"Blocking one OpenAI bot blocks them all"

OpenAI states each crawler setting is independent. GPTBot governs training. OAI-SearchBot governs ChatGPT search.

• Reverse DNS verification: the DNS check behind most bot verification.
• SEO bots vs AI crawlers: why Googlebot and AI retrieval bots need separate decisions.
• Why robots.txt allows AI bots but you still miss ChatGPT: the usual blockers beyond robots.txt.
• Free way to verify a real AI crawler hit: check a log line yourself.

Can a scraper fake being GPTBot or ClaudeBot? It can fake the name. The user-agent header is written by the client, so any script can claim to be GPTBot. It cannot easily fake the source IP within the operator's published range, a reverse DNS record on the operator's domain, or a valid cryptographic signature. That is why firewalls and log reviews should check one of those proofs before trusting or blocking a request labelled as an AI crawler.

Should I allowlist AI crawlers by IP address only? OpenAI advises against IP-only allowlisting, even though it publishes IP ranges for its crawlers. Ranges change, and an IP rule alone ignores which bot is asking. The safer setup combines the user-agent with your CDN or firewall's verified-bot programme, which keeps up with operator lists for you. Then test the result, since a rule that looks right can still return a 403.

How do I know if my firewall blocks verified AI crawlers? Request a page using each crawler's user-agent and look at the status code, plus any challenge or CAPTCHA page in the response. A 200 with real content is a pass. A 403, 429 or challenge page means the crawler is being turned away. Your own server or CDN logs, filtered to verified bots, show the same thing from the real crawler's side.

Does being a verified bot affect how AI answers describe my brand? No. Verification is an access question about who is fetching your pages. What an engine says about you depends on retrieval and generation, which you do not control. Verification matters because blocking a real crawler removes your pages from that engine's candidate set, so the answer gets written from other sources instead.