Can rate limiting stop AI crawlers from reading my site?

Rate limiting can stop AI crawlers from reading your site, and it does it silently. When a server or CDN returns 429 Too Many Requests, the response holds no page content. The crawler that asked gets nothing to store or quote.

That makes a 429 functionally similar to a block for that request. OpenAI's guidance on making a site eligible for ChatGPT search treats access as two conditions at once: allow OAI-SearchBot in robots.txt, and confirm that your host or CDN lets traffic from OpenAI's published searchbot IP addresses through. A rate limit sits in the second condition. Your robots.txt can say allow while your edge says not now.

Three things make this easy to miss:

• It's partial. Limits trigger on bursts, so some URLs get through and others don't. The ones that fail are often the deep pages a crawler reaches last.
• It's invisible from outside. No AI platform reports which of your pages it failed to fetch.
• It looks like good hygiene. Limits are set to protect the origin, and nobody checks which bots they caught.

The next move is to stop guessing and read the logs.

Not every 429 matters. It depends on which bot got it and what that bot does. As of October 2026, the major platforms run separate crawlers for training, search indexing and user-triggered fetches, and OpenAI states that each setting is independent of the others.

• Retrieval and search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, bingbot, Googlebot): a 429 here can keep a page out of the index that AI answers draw on. This is the case to fix.
• Training bots (GPTBot, ClaudeBot, Google-Extended): a 429 affects what may enter future training data. Throttling these has no documented effect on search answers.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User): these fetch a page live when someone asks. A 429 means that answer is written without your page.

There's a second-order risk too. Anthropic warns that blocking its crawlers at the IP level impedes its ability to read your robots.txt file. If your limit catches requests for robots.txt itself, the crawler can't even see your rules. The free robots.txt reachability check tests exactly that.

What you can't control: whether a page that's fetched gets retrieved for a given question, or cited. Removing a 429 restores eligibility. It doesn't promise an outcome.

Your origin or CDN logs are the ground truth. No outside tool sees your firewall decisions the way they do.

1. Pull a recent window of logs from the layer that enforces the limit. If Cloudflare or another CDN applies the rule, the origin log won't show the 429, because the request never arrived.
2. Filter by AI user agents. Search for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Meta-WebIndexer and bingbot.
3. Group by status code. Count 200, 403, 429 and any challenge pages per bot. A challenge page served with a 200 is still a failure, because the crawler got a CAPTCHA, not your content.
4. List the URLs that got 429s. Check whether pricing, product and comparison pages are among them. Those are the pages AI answers about your business depend on.
5. Check the timing. Clusters of 429s at the same minute point to a burst threshold. Steady 429s point to a limit set too low for that bot.

Our playbook on reading server logs for AI crawler status codes walks through the filters in more detail. If you don't have log access, the free AI bot log check explains other ways to see which bots visit.

The goal isn't to remove limits. It's to stop one rule from treating a search crawler like a scraper.

Verify before you exempt

Anyone can send a request with "OAI-SearchBot" in the user-agent header. Cloudflare's verified-bot criteria accept a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. OpenAI publishes its searchbot IP ranges at openai.com/searchbot.json, and also warns against relying on IP-only allowlisting. Combine user agent with a verification method.

Separate the rules by purpose

• Give verified retrieval bots their own rule with a higher threshold than anonymous traffic.
• Throttle training crawlers harder if server load is your real concern. That choice doesn't touch search eligibility.
• Never rate limit requests for /robots.txt.
• Exempt or raise limits on the URLs that carry your key facts.

Watch for defaults you didn't set

From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Settings can change without anyone on your team editing them. Our guide to when your CDN or firewall blocks AI crawlers covers the vendor settings, and Cloudflare's AI crawler settings has its own page.

Start with the pages that matter, not the whole site.

1. Run a request as each AI crawler against your pricing and product pages. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
2. Pull a week of CDN logs and count 429s per AI user agent.
3. Move verified retrieval bots into their own rate-limit rule.
4. Re-check after each CDN or firewall change, since defaults can shift under you.

If robots.txt already allows these bots and you still never appear, rate limiting is one of several causes. See what else can block you when robots.txt allows AI bots.