How do I see which AI bots are crawling my site?

Your raw access logs are the record of every request that hit your site. That includes AI crawlers. Your analytics dashboard is not, and the reason is mechanical.

Most analytics tools count a visit when a JavaScript tag runs in the browser. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but "don't execute them." No script runs, so no visit is recorded. The bot was there; your dashboard just never heard about it.

So the method is:

1. Export access logs from your web server, host or CDN (Cloudflare, Fastly, your load balancer).
2. Filter the user agent field for known AI crawler names.
3. Group by bot, URL and status code.
4. Verify that the requests really came from the operator before you change anything.

Our playbook on reading server logs for AI crawler status codes walks through the filtering step by step. If you don't have log access, there's also a free way to see which AI bots crawl your site.

Each AI company runs more than one bot, and each bot has a different job. As of October 2026, the operators document these:

The split matters. OpenAI says its settings are independent: GPTBot governs training use, while OAI-SearchBot decides whether you can appear in ChatGPT search answers. Seeing GPTBot in your logs tells you nothing about search eligibility. If that's your situation, see why your log shows GPTBot but not OAI-SearchBot.

Google is different. AI Overviews and AI Mode rely on Googlebot, and there is no separate robots.txt token for them. You won't find a distinct "Google AI" bot in your logs.

The user agent tells you who knocked. The status code tells you whether they got in.

• 200: the bot received the page.
• 403: something refused it, usually a firewall or bot-management rule. OpenAI warns that these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."
• 429: rate limiting. The bot got throttled.
• 200 with a challenge page: the response is a CAPTCHA or JavaScript check, not your content. Automated crawlers fail these.

This is the failure most sites never notice. Your robots.txt allows a bot, but your CDN turns it away. OpenAI states that eligibility for ChatGPT search needs both: allow OAI-SearchBot in robots.txt and let traffic from its published IP addresses through your host or CDN.

CDN defaults can change without you touching anything. Since 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers. If you're on Cloudflare, check whether its AI crawler settings are hiding your site.

A user agent is just a text string the client sends. Anyone can send "GPTBot" in a request. Scrapers do this to slip past rules that trust AI crawlers. So a log line that says OAI-SearchBot proves nothing on its own.

Verify with one of these:

• Published IP ranges. OpenAI publishes JSON files of IP ranges for its bots, including one for OAI-SearchBot. Match the request's source IP against the list.
• Reverse DNS. Look up the hostname for the IP, then confirm it resolves back to the same IP and belongs to the operator's domain.
• Your CDN's verified-bot flag. Cloudflare marks a bot as verified when it has a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or valid reverse DNS.

Don't overcorrect into IP-only allowlisting. OpenAI publishes its ranges but also warns against relying on IP alone. Anthropic notes that IP blocking can stop it reading your robots.txt at all. Combine user agent and verification, then decide.

Logs show access. They don't show outcomes.

• Visits are not citations. A bot can fetch every page and the engine can still never name you. Retrieval and citation happen later, and you don't control them. See why AI crawlers visit but AI never cites you.
• Answer-time fetchers behave differently. ChatGPT-User and Perplexity-User fetch pages when a user asks. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." The difference between index-time crawlers and answer-time fetchers explains why.
• A 200 isn't the whole story. If key content only appears after JavaScript runs, the bot got a page but not your content.

1. Pull 30 days of logs and list every AI user agent with its status codes.
2. Verify the top agents by IP range or reverse DNS.
3. Look for search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer) getting 403s, 429s or challenges. Fix those rules at the CDN or firewall.
4. Confirm your robots.txt rules per bot match what you intend.

If you want to test reachability without waiting for a crawler to visit, the AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Why doesn't Google Analytics show AI bot traffic? Analytics tools count a visit when their JavaScript tag runs. Vercel's study found ChatGPT and Claude crawlers download JavaScript files but don't execute them, so the tag never fires. The request still lands on your server, which is why raw access logs or CDN logs are the place to look. Analytics can show human referrals from AI apps, but not the crawler visits themselves.

Should I block GPTBot if I see it in my logs? That depends on whether you want your content used for model training. GPTBot governs training only. Blocking it does not remove you from ChatGPT search, because OAI-SearchBot controls that, and OpenAI states the settings are independent. Decide on training use separately from search, and set robots.txt rules per bot rather than with one blanket AI block.

How can I tell a real AI crawler from a fake one? Check the source IP. OpenAI publishes IP ranges for its bots, and a real OAI-SearchBot request should come from one of them. Reverse DNS works for operators that support it: the hostname should belong to the operator and resolve back to the same IP. Many CDNs, including Cloudflare, also flag verified bots, which saves you doing the lookups by hand.

My logs show AI bots getting 403 errors. What does that mean? A 403 means something refused the request, usually a firewall, WAF or bot-management rule. Your robots.txt may allow the bot while your CDN blocks it. For a search bot like OAI-SearchBot, that silently removes your eligibility for ChatGPT search answers. Find the rule that matched, confirm the requests were genuine, and add an exception for verified crawlers you want to allow.