Crawl-to-referral ratio: definition and caveats

The crawl-to-referral ratio is crawler requests divided by referred visits, for one platform, over one time window.

• Numerator: requests from that operator's bots, counted from server or CDN logs.
• Denominator: human visits that arrive from that operator's product, counted in analytics.

A ratio of 100:1 means the platform fetched 100 pages for every visitor it sent. The number says nothing on its own about whether the platform cites you, values you, or gets your facts right. It is a traffic-exchange measure, not a visibility score.

Each AI operator runs several bots with different jobs, and only some of them feed a product that can send a visitor.

• Training crawlers collect content for model training. OpenAI says GPTBot crawls content "that may be used in training" its foundation models. ClaudeBot and Google-Extended play similar roles for Anthropic and Google. A training crawler has no search results page, so its referral count is close to zero by construction.
• Search crawlers build an index that answers can link to. OpenAI says OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features." PerplexityBot and Claude-SearchBot do the equivalent work for their platforms.
• User-triggered fetchers such as ChatGPT-User and Perplexity-User fetch a page because a person asked.

Blend these into one numerator and the ratio mostly tells you how much training crawling the operator does. That is why a platform heavy on training looks extreme and a search-led one looks mild. To see which bots hit you, check which AI bots are crawling your site and split the log by user agent before you divide anything.

The ratio is useful for one decision: whether a given bot's load is worth what you get back. It is a poor guide to whether you show up in AI answers.

The denominator is incomplete. Google AI Overviews emit no referral at all, so their visits cannot be counted as AI referrals. ChatGPT tags some links with a UTM parameter, but free-tier and mobile-app clicks often arrive untagged. Read how AI referral traffic is measured before trusting any denominator.

The policy risk is real too. Blocking a bot because its ratio looks bad can remove you from a product that does refer. Blocking GPTBot stops training use only. OpenAI states each setting "is independent of the others," and OAI-SearchBot is the one that governs whether ChatGPT search can show your pages. The next move: compute the ratio per bot, then decide per bot.

"Platform X crawls N pages per referral"

Published ratios are snapshots. A widely quoted Anthropic figure came from a July 2025 Cloudflare snapshot. In the evidence we hold, 2026 figures are roughly an order of magnitude lower across every platform, and Perplexity's direction reversed. Do not quote an old ratio as current.

"A low ratio means the platform is generous"

It mostly means the operator sends a smaller share of training crawls. Purpose explains the gap better than generosity.

"More crawling means more visibility"

Crawl volume is not citation. A bot can fetch a page often and never cite it. Crawl frequency also varies by bot, which this comparison of AI and Google crawl frequency covers.

"Zero referrals means the bot is blocked"

Not necessarily. A firewall or CDN can block an allowed crawler, and the reverse also happens. If counts suddenly drop, check whether Cloudflare's AI crawler settings changed your access.

• AI referral traffic: the denominator, and its blind spots.
• UTM parameters: how some AI platforms tag outbound links.
• Measuring AI referral traffic in analytics: the setup that gives you a usable denominator.
• Training crawler, search crawler, user-triggered fetcher: the three bot types that make up the numerator.

How do I calculate the crawl-to-referral ratio for my own site? Count requests per bot user agent from your server or CDN logs over a fixed window, for example 30 days. Count referred sessions from the matching platform in analytics over the same window. Divide requests by sessions, separately for each operator. Keep training bots apart from search bots, or the result mostly measures training activity. Note the date, because the ratio will change.

Should I block crawlers with a high ratio? Decide per bot, not per ratio. Blocking a training crawler such as GPTBot stops training use and, per OpenAI, does not affect ChatGPT search. Blocking a search crawler such as OAI-SearchBot removes you from that product's answers. A high ratio for a training bot is expected. A high ratio for a search bot is worth investigating before you act.

Why are published ratios so different from each other? They measure different networks, windows and bot mixes. Some count every bot an operator runs. Others count only search fetchers. Referral counting also varies, because some AI surfaces tag links and others send no referral signal. A ratio from one network in one month is a snapshot of that sample, not a property of the platform.

Does the ratio tell me whether AI platforms cite my brand? No. It compares bot load with visits. A platform can cite you in answers that produce no click, and Google AI Overviews send no referral at all. To know whether you are named, you need to look at the answers themselves, engine by engine, rather than at traffic.