Reverse DNS verification for crawlers

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

Reverse DNS verification is a check that confirms a crawler's IP address resolves to a hostname on the operator's domain, and that the hostname resolves back to the same IP. It lets a server tell a real Googlebot or AI crawler from a bot that only copies its user-agent string. It sits alongside published IP lists as one of the accepted ways to verify a bot.

Key takeaways

Reverse DNS verification is a two-way lookup: IP to hostname, then hostname back to the same IP.
A user-agent string proves nothing on its own, because anyone can send any user agent.
Some operators support DNS-based checks and others publish IP lists, so you match the method to the crawler.
Verification protects you from spoofed bots, but a firewall that misclassifies a real crawler can silently remove you from AI answers.
Passing verification confirms identity and reachability only, not crawling frequency, indexing or citation.

What reverse DNS verification means

Reverse DNS verification answers one question: is this request really from the company it claims to be from? A request header can say Googlebot or OAI-SearchBot, but that string is typed by whoever sends the request. The IP address is harder to fake, and DNS lets you tie that IP back to a domain the operator controls.

Cloudflare lists reverse DNS as one of the ways a bot can qualify as verified, next to a cryptographic Web Bot Auth signature and a published IP list with a stable user agent. So reverse DNS is one method among several, not the only test. For the wider concept, see what a verified bot is.

How a reverse DNS check works

The check has two halves, and both must pass.

Reverse lookup (rDNS lookup). Take the IP address from your server log and ask DNS for its hostname (the PTR record).
Domain check. Confirm the hostname ends in a domain the crawler's operator documents for that bot. Use the operator's own crawler page for the list, not a forum post.
Forward lookup. Resolve that hostname back to an IP address.
Match. The forward result must equal the original IP. If it does, the request came from the operator.

The forward step matters because whoever controls an IP range controls its PTR record. A spoofer can point their own IP at a convincing hostname. They can't make the operator's DNS resolve that hostname back to their IP.

Not every AI crawler is built for this method. As of October 2026, OpenAI publishes its crawler IP ranges as JSON files, for example the OAI-SearchBot list at openai.com/searchbot.json. For those bots you compare the IP against the published ranges instead. The playbook on verifying real AI crawlers covers each operator's method in depth.

Why it matters for AI visibility

Most sites meet verification from the other side: their firewall or CDN does it for them. That is where visibility breaks quietly.

OpenAI's help documentation says ChatGPT search eligibility needs two things: robots.txt must allow OAI-SearchBot, and the host or CDN must allow traffic from OpenAI's published searchbot IP addresses. A firewall that fails to recognise a real crawler returns a 403, and nothing in your dashboards tells you. Anthropic warns that IP blocking also stops its crawler from reading your robots.txt at all.

Defaults can change without you editing anything. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed.

The next move is to check both gates. Confirm your robots.txt rules for each AI bot, then confirm the firewall actually lets those bots through. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. To check individual log entries yourself, use the free AI crawler verification method.

Common misconceptions

"The user agent says Googlebot, so it's Googlebot." User agents are self-declared. Verification exists because they are trivial to copy.
"A reverse lookup alone is enough." Without the forward lookup, a PTR record you don't control proves nothing.
"IP allowlisting is the safest setup." OpenAI publishes IP ranges but also warns against relying on IP-only allowlisting. Pair the IP check with the user agent and your CDN's verified-bot programme.
"All crawler IP lists cover IPv6." OpenAI's searchbot.json held IPv4 prefixes only when the evidence base checked it. An edge that answers only over IPv6 can't be reached by those crawlers.
"A verified crawler will index or cite me." Verification proves identity and lets the request through. What the engine indexes, retrieves or cites is outside your control.

Frequently asked questions

How do I verify Googlebot from my server logs?

Take the IP from the log line and run a reverse DNS lookup to get its hostname. Check that the hostname belongs to a domain Google documents for its crawlers. Then resolve that hostname forward and confirm it returns the original IP. If any step fails, treat the request as an impostor using Googlebot's name, and handle it like any other unverified bot.

Should I use reverse DNS or a published IP list?

Use whichever method the operator documents for that crawler. Some support DNS-based checks. Others, such as OpenAI as of October 2026, publish JSON files of IP ranges for each bot. Cloudflare accepts either as a basis for verified-bot status. Where both exist, an IP list is faster because it avoids two DNS queries per request, but the list must be refreshed as it changes.

Can reverse DNS verification block a real AI crawler by mistake?

The check itself is reliable when done in full. Mistakes come from configuration: a firewall rule that expects DNS verification from a crawler that only publishes IP ranges, a stale IP list, or an IPv6-only edge. OpenAI notes bot-mitigation systems can block legitimate crawlers, often with 403 errors. Test each crawler's access rather than assuming your rules work.

Does passing verification mean my content appears in AI answers?

No. Verification only confirms the request is genuine and lets it through. After that, the engine still decides what to index, which pages to retrieve for a query, and what to cite. Blocking a real crawler can remove you from an engine's answers, so verification protects access. It does not create visibility beyond that.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.