What is crawl budget?

Crawl budget is the number of URLs a crawler will fetch from your site in a given period. It is a limit set by the crawler, not a setting you choose.

Two inputs decide it:

• Crawl capacity (also called the crawl rate limit): how many requests the crawler sends, and how fast, without slowing your server. Slow responses and server errors push it down.
• Crawl demand: how much the crawler wants your URLs. Popular pages and pages that change often attract more fetches.

The term comes from classic search, where Googlebot is the main example. It applies loosely to any crawler that has to decide which URLs to fetch first.

A crawler keeps a queue of URLs for your site. Each visit, it fetches some of them, within the rate your server tolerates. URLs it doesn't reach wait for the next pass.

The budget gets wasted when the queue fills with URLs nobody needs:

• Faceted navigation and URL parameters that create near-duplicate pages
• Long redirect chains
• Soft 404s and error pages that return 200
• Session IDs or tracking parameters in internal links

On a site with a few hundred pages, none of this usually matters. The crawler gets through everything. On a site with hundreds of thousands of URLs, such as a large retailer or a listings site, waste can delay new pages from being fetched at all.

Crawl budget is separate from the per-URL fetch ceiling. Google's March 2026 post on Googlebot says it fetches up to 2MB of any individual URL (excluding PDFs) and ignores bytes after that. If you suspect a page hits that limit, see whether your page is too large for Google and the payload limit check.

Less than most advice suggests. As of October 2026, no AI vendor publishes how its crawlers allocate fetches across a site, and nothing in the evidence measures it. Anyone quoting an "AI crawl budget" is describing a model of their own.

What the evidence does support is that access gates decide whether an AI crawler can use your page at all:

• Per-bot robots.txt permission. OpenAI documents that GPTBot (training) and OAI-SearchBot (ChatGPT search) are independent settings. Read more on whether OAI-SearchBot needs to crawl you for ChatGPT to cite you.
• Network reachability. A firewall or CDN can return 403 to a crawler that robots.txt allows. Cloudflare changed its AI crawler defaults on 2026-09-15, which is covered in whether Cloudflare could be hiding your site.
• Server-side rendering. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Content that only appears after JavaScript runs may be invisible to them, however often they visit.

If you want to see how often AI bots actually hit your pages, start with your server logs. The free AI bot log check walks through it, and how often AI crawlers visit compared with Google covers what the frequency can and can't tell you.

"Every site needs to manage crawl budget"

No. Small and mid-sized sites rarely hit it. Fix duplicate URLs because they're messy, not because you're out of budget.

"More AI crawls mean more AI citations"

No published study shows this. A crawl makes a page eligible. Whether it gets retrieved and cited for a given question is decided at answer time, outside your control.

"Blocking training bots frees budget for search bots"

Each bot is a separate crawler with its own behavior. OpenAI states its settings are independent of each other. Block or allow each one for its own consequence.

"Pinging IndexNow gets AI to crawl and cite you faster"

The IndexNow specification never mentions AI, and a 200 response only confirms the ping was received. Google does not consume it.

"noindex saves crawl budget"

A crawler has to fetch a page to see its noindex tag. A robots.txt disallow stops the fetch. They do different jobs.

To test the access gate directly, the free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

• Crawl rate: requests per second a crawler sends to your server. One half of crawl capacity.
• Crawl capacity: the load your server can handle before the crawler backs off.
• Crawl demand: how much the crawler wants your URLs, driven by popularity and change frequency.
• Crawl-to-referral ratio: pages crawled per visitor sent back. Published figures are dated snapshots. See crawl-to-referral ratio for the caveats.
• robots.txt: the per-bot file that tells crawlers which paths they may fetch.

How do I know if my site has a crawl budget problem? Look for important pages that stay unfetched for weeks while the crawler spends requests on parameter URLs, redirects or error pages. Your server logs show which URLs each bot requests and how often. Google Search Console's crawl reporting shows Googlebot's side. If your site has a few hundred pages and they all get fetched, you don't have a crawl budget problem.

Do AI crawlers have a crawl budget? Any crawler has to choose which URLs to fetch, so in practice they ration. But as of October 2026, OpenAI, Anthropic, Perplexity and others don't publish how they allocate crawls. Nobody outside those companies has measured it either. Check the things you can see: whether each bot is allowed, whether it gets a 200 response, and whether your content is in the raw HTML.

Can I increase my crawl budget? You can raise the ceiling by keeping your server fast and error-free, so crawlers don't back off. You can spend it better by removing duplicate URLs, fixing redirect chains and keeping sitemaps clean. You can't directly raise crawl demand. That depends on how much the crawler values your pages, which you influence only indirectly.

Should I block AI crawlers to protect crawl capacity for Google? Only if AI crawler traffic is genuinely straining your server, which your logs will show. Blocking has consequences. Disallowing OAI-SearchBot removes you from ChatGPT search answers, and disallowing PerplexityBot affects Perplexity's index. Decide bot by bot, and prefer rate controls at your CDN over blanket blocks if load is the real issue.