Reading AI Crawler Server Logs for Status Codes and Silent 403s

Your robots.txt says what you intend. Your logs say what actually happened. The two often disagree, and nothing else tells you.

OpenAI's help center states that eligibility for ChatGPT search needs two things: allowing OAI-SearchBot in robots.txt, and confirming that your host or CDN allows traffic from OpenAI's published searchbot IP addresses. The same OpenAI guidance warns that bot-mitigation systems can mistakenly block legitimate crawlers, often returning 403 Forbidden errors. That second gate is the one most sites never test.

Analytics tools won't help here. Crawlers don't run your tracking script, which is why AI crawler visits don't show up in Google Analytics. Logs are the only record.

Edge logs versus origin logs

Where you look matters more than how you look.

• CDN or WAF logs record every request, including the ones your firewall blocked or challenged.
• Origin server logs only record requests that made it past the edge.

If your CDN returns a 403 to PerplexityBot, your origin log shows nothing at all. An empty origin log can mean "never visited" or "blocked upstream". Only the edge log tells you which. Start there whenever you sit behind a CDN.

Each crawler token belongs to one operator and one purpose. Blocking one has a different consequence from blocking another. As of October 2026, the operators document these tokens:

OpenAI's crawler page says each setting is independent of the others. So seeing GPTBot in your logs tells you nothing about OAI-SearchBot. If one shows up and the other doesn't, read why logs show GPTBot but not OAI-SearchBot.

This works on any access log in the common combined format. Adjust field positions if your CDN exports JSON or a custom layout.

1. Export the right logs. Pull 7 to 30 days from your CDN or WAF first, then from your origin. Include the status code, user agent, client IP, path and response size.
2. Filter by user agent. Match each token from the table above, case-insensitive.
3. Count status codes per agent. Don't lump all AI bots together. A healthy GPTBot line can hide a broken OAI-SearchBot line.
4. Check robots.txt requests separately. Filter for the path /robots.txt. Anthropic warns that IP blocking impedes its ability to read your robots.txt file, so a blocked robots.txt fetch is its own failure.
5. Verify the requester. Anyone can send a request with "GPTBot" in the header. Check the client IP against the operator's published list (OpenAI publishes OAI-SearchBot ranges at openai.com/searchbot.json) or use reverse DNS. Cloudflare's bot documentation describes how verified bots are identified.
6. Sample the response bodies. Pick a few 200 responses with unusually small byte counts. Open them. A challenge page or CAPTCHA served with a 200 is still a block.
7. Compare against robots.txt. Every agent your robots.txt allows should be getting 200s at the edge. Any mismatch is a finding.

For step 3, a quick count on a standard log looks like this:

grep -i "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c

Repeat it for each token. Field 9 is the status code in the combined format.

If you don't have shell access to logs, the free AI bot log check walks through other ways to see which bots arrive.

Read each agent's status mix against what your robots.txt intends.

Good

• 200 on content pages for every search crawler you allow, with response sizes close to what a browser gets.
• 304 (not modified) mixed in. That's a crawler revalidating a cached copy.
• 301 or 302 that resolve to a 200 in one hop.
• 200 on /robots.txt for every agent.

Bad

• 403 to an agent your robots.txt allows. Your WAF or bot rules are overriding your stated policy.
• 429 in bursts. Rate limiting is throttling the crawler. See whether rate limiting can stop AI crawlers reading your site.
• 200 with a tiny payload. Usually a JavaScript challenge or interstitial, not your content.
• 5xx clustered on one agent. Often a bot rule misfiring, or an origin that times out under crawl load.
• No requests at all at the edge from an agent you allow. That's not a block, but it means nothing has fetched you yet, so there's nothing to fix in the logs.

Say you run a hypothetical online store behind a CDN. Robots.txt allows OAI-SearchBot. Edge logs show OAI-SearchBot getting 403 on every product URL while Googlebot gets 200. That is a complete, silent ChatGPT search failure, and the fix sits in the firewall rules, not the content.

One more thing to watch: crawlers that fetch .js files. OpenAI's and Anthropic's crawlers download script files without executing them, so a 200 on your JavaScript bundle doesn't mean the crawler saw rendered content. Read why AI crawlers download JavaScript they don't run if your key content only appears after hydration.

A 200 in your logs proves one thing: the crawler could fetch the page. It says nothing about whether any AI answer used, quoted or cited it.

Three reasons to keep these apart:

• Training crawlers have no referral surface. GPTBot and ClaudeBot collect pages for model training. No answer links back from a training crawl, by design. High training-crawler volume with zero referrals is expected, not a problem.
• Retrieval is per query and discretionary. A search crawler indexing your page buys eligibility. Whether the page enters a given answer is decided later by the engine.
• Crawling and citing can split completely. OtterlyAI's 2026 experiment with roughly 2,000 AI-generated posts reported that ChatGPT's crawlers kept visiting the test sites while ChatGPT citations stayed effectively at zero.

So treat logs as a gate check. They answer "can the bot get in?" They can't answer "does AI recommend us?" For that gap, see why AI crawlers visit but AI never cites you.

Run through this list on each review:

• Edge (CDN or WAF) logs exported, not just origin logs
• Every AI token from the table filtered separately
• Status code counts recorded per agent, per week
• /robots.txt requests returning 200 for every agent
• Requests verified against published IP ranges or reverse DNS
• Small 200 responses opened to rule out challenge pages
• Every allowed agent receiving 200s on key content pages
• Any 403, 429 or 5xx traced to a specific firewall, rate-limit or bot rule
• Robots.txt rules re-read to confirm they match intent (the robots.txt checker for AI crawlers parses them per agent)
• Training bots and search bots judged separately, never as one "AI traffic" number

If you can't get edge logs, test from the outside instead. The firewall AI bot test shows whether your WAF turns crawlers away, and 99Visibility's AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall rejects.

Access breaks without anyone touching your site. Plan for that.

• Monthly as a baseline, using 30 days of edge logs.
• After any CDN, WAF or bot-management change, within a week. Rule updates are the most common cause of a new 403.
• After your CDN changes its defaults. From 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stays allowed. A site can lose access without its owner editing anything.
• After a site migration, redesign or hosting move. New infrastructure often ships new bot rules.

Keep the per-agent counts in a simple sheet. A trend in status codes is far easier to read than a single snapshot. Once access is clean, the next step is understanding which queries actually retrieve you, which Bing exposes through its grounding query data.

Why do my origin logs show no AI crawler traffic at all? Either no AI crawler has requested your pages, or your CDN or firewall blocked them before they reached your server. A request rejected at the edge never appears in the origin log. Check your CDN or WAF logs for the same period. If those show 403s or challenges to AI user agents, the block is upstream. If those are also empty, the crawlers simply haven't fetched you yet.

Should I worry about a 403 to GPTBot? Only if you want your content available for OpenAI model training. GPTBot governs training use, and OpenAI states each crawler setting is independent. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot. A 403 to OAI-SearchBot is the one that matters for ChatGPT search answers. Decide each agent on its own purpose.

Can I trust the user agent string in my logs? Not on its own. Any client can claim to be GPTBot or PerplexityBot. Confirm the request comes from the operator by matching the IP against published ranges, such as OpenAI's searchbot.json file, or by reverse DNS. Spoofed bot traffic can inflate counts or trigger rate limits that then catch the real crawler too.

Do Perplexity-User and ChatGPT-User follow my robots.txt? Not reliably. Perplexity's documentation says Perplexity-User generally ignores robots.txt rules, and OpenAI says robots.txt may not apply to ChatGPT-User because a person triggers those fetches. If you see these agents in your logs despite a disallow rule, that is documented behavior. Firewall rules are the control if you need to stop them.

Does a high crawl count mean AI is recommending my brand? No. Crawl volume measures fetching, not answers. Training crawlers never send referral traffic by design, and search crawlers only make you eligible for retrieval. To see what AI platforms actually say about you, you need to look at the answers themselves. The free checker on this site shows whether engines name you for up to three buyer questions.