Free AI Crawler Verification: How to Check a Real AI Crawler Hit

99Visibility Research · Updated October 21, 2026 · 6 min read

Short answer

To verify an AI crawler hit for free, take the IP address from your server log and check it against the operator's published IP list, such as OpenAI's searchbot.json for OAI-SearchBot. Where an operator documents a hostname, confirm it with a reverse DNS lookup followed by a forward lookup. A user agent string alone proves nothing, because anyone can send it. OpenAI also warns against IP-only allowlisting.

Key takeaways

A user agent string is self-declared, so a log line saying OAI-SearchBot is a claim until the IP checks out.
OpenAI publishes separate IP files per crawler, including searchbot.json for OAI-SearchBot and gptbot.json for GPTBot.
Reverse DNS only counts when the forward lookup of the returned hostname resolves back to the same IP.
OpenAI warns against IP-only allowlisting, so pair IP checks with user agent matching and your CDN's verified-bot settings.
Verifying a hit tells you who visited, not whether your firewall let them through or whether any engine will cite you.

What the free check tells you

A crawler verification answers one question: did this request really come from the operator it claims? Your log shows a user agent like OAI-SearchBot or GPTBot. That string is just text the client sends, and scrapers copy it freely.

The check matters in two directions:

Fake hits inflate your picture. If a scraper spoofs OAI-SearchBot, you may think ChatGPT search is crawling you when it isn't.
Real hits get blocked by mistake. If you block anything that looks like a bot, you may turn away the real crawler that controls your ChatGPT search eligibility.

Each OpenAI bot has a different job. OpenAI's crawler page says GPTBot governs training use, OAI-SearchBot governs ChatGPT search, and "each setting is independent of the others". So verify the specific bot, not "OpenAI" in general. Our GPTBot definition covers what that one controls. For the full method across operators, see the playbook on verifying real AI crawlers vs spoofed user agents.

How to verify a hit by hand

You need shell access to your logs, or a log export. Say your access log shows a request with the OAI-SearchBot user agent.

Pull the IPs claiming the bot. In a common log format file, the IP is the first field.

grep "OAI-SearchBot" access.log | awk '{print $1}' | sort -u

Download the operator's published list. OpenAI publishes one file per crawler: https://openai.com/searchbot.json for OAI-SearchBot, plus gptbot.json, chatgpt-user.json and adsbot.json at the same path. List the prefixes:

curl -s https://openai.com/searchbot.json | jq -r '..|.ipv4Prefix? // empty'

Test each IP against the ranges. Python's standard library does the CIDR math. Replace the example address with yours.

python3 -c "import ipaddress,json,sys,urllib.request as u; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(u.urlopen('https://openai.com/searchbot.json')); print(any(ip in ipaddress.ip_network(p['ipv4Prefix']) for p in d['prefixes'] if 'ipv4Prefix' in p))" 203.0.113.7

Use reverse DNS where the operator documents a hostname. Look up the PTR record, then look up the returned hostname and confirm it points back to the same IP.

dig -x 203.0.113.7 +short
dig +short crawler-host.example.com

Check the bot's own documentation for the method it supports. OpenAI's answer is the IP files. Perplexity documents its crawlers, PerplexityBot and Perplexity-User, on its own crawler page. Our glossary entry on reverse DNS verification explains the two-step lookup in more detail.

If you haven't yet listed which bots appear in your logs at all, start with the free AI bot log check.

How to read the result

There are three outcomes.

User agent matches and the IP sits in the published range. Treat it as the real crawler. If it got a 200, that bot can fetch the page.
User agent matches but the IP is outside every published range. Treat it as unverified. It is likely a spoofer, though it could also be a range the operator added after you downloaded the file. Re-fetch the list before you conclude.
Reverse DNS returns a hostname that doesn't forward-confirm. Treat it as fake. Anyone who controls an IP block can set a PTR record to any name they like.

Then check the status code your server returned to verified hits. A verified OAI-SearchBot getting a 403 or a challenge page is the expensive finding. OpenAI's help article says eligibility for ChatGPT search means allowing OAI-SearchBot in robots.txt and confirming your host or CDN allows traffic from its published IP addresses. Robots.txt alone is not enough behind a blocking firewall. Our page on OpenAI crawlers, robots.txt and firewalls walks through both halves.

Why IP checks alone are not enough

OpenAI publishes its IP ranges and also warns against IP-only allowlisting. The practical reading: use the IP list to verify, but build your allow rules on user agent plus a verified-bot programme, not on a hand-copied list of prefixes.

Cloudflare's bot documentation names the signals it accepts for a verified bot: a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, plus robots.txt compliance. If you use Cloudflare or a similar CDN, its verified-bot setting does this matching for you and updates as operators change ranges.

The next move: check your CDN's bot settings, not just robots.txt. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can lose AI crawler access without anyone editing a file. To confirm your robots.txt rules separately for each bot, follow how to test robots.txt rules for each AI bot.

Limits of the check

Be plain about what this proves and what it doesn't.

Lists change. Each OpenAI file carries a creationTime. As of the evidence check on 22 August 2026, chatgpt-user.json was dated 14 August 2026. A list you saved months ago can mark a real bot as fake.
IPv4 only, for now. As of 22 August 2026, OpenAI's four IP files held zero IPv6 prefixes. An origin reachable only over IPv6 can't be reached by those crawlers at all.
User-triggered fetchers play by different rules. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." A verified hit from these is a user request, not an index crawl.
A verified visit is not a citation. Crawling makes a page eligible. Retrieval for a given question and citation in the answer are decided by the engine, and nobody outside it controls those stages.
Your logs only show requests that reached your server. If your CDN blocked a bot at the edge, your origin log may never see it. Check the CDN's own logs too.

The faster way to test crawler access

Log verification looks backwards: it tells you who already visited. The complementary check looks forward: request your page as each AI crawler and see what your stack returns.

99Visibility's free AI Crawler Checker does that second job. It requests a page as each AI crawler and shows which ones your firewall turns away. Use it alongside the log check, not instead of it. Logs tell you whether the real OAI-SearchBot is arriving. The request test tells you what it would get if it did.

If both come back clean, the access gate is open for that bot. What the engine does next is outside your control.

Frequently asked questions

Can I verify GPTBot the same way as OAI-SearchBot?

Yes. OpenAI publishes a separate IP file for each crawler, so check GPTBot hits against gptbot.json and OAI-SearchBot hits against searchbot.json. Don't mix them up. GPTBot governs training use only, while OAI-SearchBot controls whether you can appear in ChatGPT search answers. Blocking one does not block the other, because OpenAI states each setting is independent.

Why would someone fake an AI crawler user agent?

Scrapers send well-known crawler names hoping site owners allow them through rules written for legitimate bots. Since the user agent is plain text the client chooses, faking it costs nothing. That's why verification rests on something the spoofer can't easily copy: an IP address inside the operator's published range, or a reverse DNS name that forward-confirms to the same address.

Is a reverse DNS lookup on its own enough?

No. A reverse lookup returns whatever PTR record the owner of that IP block set, and anyone can set a misleading one. You need the second step: resolve the returned hostname forward and confirm it points back to the original IP. Use this method only for operators that document a hostname pattern. For OpenAI, use the published IP files.

How often should I re-download OpenAI's IP lists?

Re-download before every verification run rather than relying on a saved copy. Each file includes a creationTime field that shows when it was last generated, and the lists do change. A stale copy can flag a real crawler as fake. If you allow crawlers at the firewall, rely on your CDN's verified-bot setting, which tracks changes for you.

If verified OpenAI bots get 200s, will ChatGPT cite my site?

Not necessarily. A 200 to a verified OAI-SearchBot means the access gate is open, which is the part you control. Whether ChatGPT searches for a given question, retrieves your page and cites it is decided by the engine. Nobody outside OpenAI controls or measures that directly. Access is necessary, not sufficient.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.