Free AI Bot Crawl Check: See Which AI Bots Crawl Your Site

A log check answers one question: did a named AI crawler request your pages, and what did your server send back? That is a first-party fact. No third-party tool can see it without your logs.

It tells you three useful things:

• Which bots showed up. Training crawlers, search crawlers and user-triggered fetchers each have their own user agent.
• What they received. The status code per request shows whether a bot got content or got blocked.
• Where they went. The request paths show whether bots reached the pages that matter, such as pricing or product pages.

This matters because access is the gate before everything else. A page that a search crawler can't fetch can't enter that engine's index, whatever its content says.

Google Analytics and similar tools count a visit when a JavaScript tag runs in a browser. Most AI crawlers never run that tag. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. A bot that doesn't execute JavaScript also never fires an analytics tag.

So an empty AI segment in analytics does not mean no AI bots came. It means your analytics can't see them. The longer explanation is in why AI crawler visits don't show up in Google Analytics. The record that does capture them is the raw request log on your origin server or your CDN.

You need shell access to an access log in the common combined format (Nginx and Apache default to it), or a log export from your CDN.

1. Find the log. Typical paths are /var/log/nginx/access.log or /var/log/apache2/access.log. On a CDN, export request logs for the last 7 to 30 days from its dashboard.
2. Filter for AI user agents. Search for the crawler names the operators publish:

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Meta-WebIndexer|bingbot" access.log > ai-bots.log

3. Count status codes per bot. In the combined format the status code is field 9:

for b in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do echo "$b"; grep -i "$b" ai-bots.log | awk '{print $9}' | sort | uniq -c; done

4. See which pages each bot requested. Run grep -i "OAI-SearchBot" ai-bots.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20.
5. Check robots.txt fetches. Run grep -i "robots.txt" ai-bots.log | awk '{print $9}' | sort | uniq -c. A bot that can't read robots.txt can't read your rules either.
6. Spot-check the source IPs of a few hits against the operator's published ranges. OpenAI publishes its search crawler ranges at openai.com/searchbot.json. The method is in how to verify a real AI crawler hit.

There is no separate crawler for Google AI Overviews or AI Mode. Googlebot handles all Google Search surfaces, so look for Googlebot there.

Read each bot on its own line. One healthy bot says nothing about the others.

OpenAI's own guidance notes that bot-protection systems can mistakenly block legitimate crawlers, often returning 403 errors. A 403 to OAI-SearchBot is a complete, silent failure for ChatGPT search.

The most common misreading is GPTBot. OpenAI says GPTBot crawls content that may be used for training, and OAI-SearchBot surfaces sites in ChatGPT search. Its documentation states that each setting is independent of the others. So a log full of GPTBot hits tells you nothing about ChatGPT search eligibility. If you see GPTBot but not OAI-SearchBot, check your robots.txt rules for OAI-SearchBot specifically. Anthropic and Perplexity split their bots the same way: ClaudeBot is for training, Claude-SearchBot for search, and PerplexityBot for Perplexity's search index.

Logs are strong evidence of access. They are weak evidence of everything after it.

• Blocked requests may never reach your origin. If a CDN or firewall rejects a bot, the refusal lives in the CDN's logs, not your server's. Check both. As of September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers, so a site can lose access without anyone editing a setting.
• User agents can be spoofed. Anyone can send a request labelled GPTBot. Verify important hits by IP.
• Absence is ambiguous. A bot may simply not have visited in your window. Testing with a firewall access test tells you whether it could.
• User-triggered fetchers play by different rules. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules.
• A 200 is not a citation. Logs can't show retrieval for a real user's question or whether an answer used your page. That gap is why AI crawlers can visit and still never cite you.

Logs show what already happened. A reachability test shows what happens now, and it works even without log access. The free AI crawler checker requests a page as each AI crawler and shows which ones your firewall turns away. Run it alongside your log check: if logs show 403s and the checker confirms the block, the fix sits in your CDN or firewall rules.

If the block turns out to be in robots.txt instead, start from a robots.txt template for AI search and training bots that allows search crawlers per bot and makes a separate decision about training crawlers.