If I block an AI crawler's IPs, can it still read my robots.txt?

A crawler has to reach your server before it can read anything, including /robots.txt. If your firewall, CDN or server drops requests from a crawler's IP ranges, the crawler can't fetch the file. Your rules exist, but nobody on the other side reads them.

Anthropic states this directly: IP blocking "impedes our ability to read your robots.txt file." That has two consequences.

• Your stated rules stop governing. You may have written a careful per-bot policy, such as allowing Claude-SearchBot while disallowing ClaudeBot. An IP block makes that policy irrelevant, because the network decides first.
• You lose precision. robots.txt can treat each bot differently. An IP block usually hits everything an operator runs from those ranges, so it can remove you from search surfaces you meant to keep.

The practical rule: say what you want in robots.txt, and make sure every crawler can actually fetch that file. For background on how Anthropic's three agents differ, see ClaudeBot, Claude-SearchBot and Claude-User explained.

Think of AI crawler access as two gates in sequence.

1. The network gate. Your host, CDN, WAF or bot-management rules decide whether the request gets a response at all.
2. The robots.txt gate. If the request gets through, the crawler reads your rules and follows them for its own user agent.

OpenAI describes the same structure for ChatGPT search. Its help page says that to make a site eligible, you allow OAI-SearchBot to crawl and confirm that your host or CDN allows traffic from OpenAI's published searchbot IP addresses. The word is "and". robots.txt alone is not enough behind a blocking firewall.

The network gate can also close without anyone on your team touching it. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. A site in that population can lose AI crawler access while its robots.txt still says allow. The full comparison of both gates is in robots.txt vs firewall for AI crawlers.

The evidence here is first-party: the companies that run the crawlers describe how their bots behave. That's a documented mechanism. How often IP blocks override robots.txt across the web has not been measured.

OpenAI also notes that bot-mitigation systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." That's the failure mode worth checking for: nothing in your analytics flags it, and the crawler simply stops getting content. Anthropic's full guidance is summarised in Anthropic's crawler documentation, digested.

User-triggered fetchers

Some agents fetch a page because a person asked for it. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. For these agents, robots.txt was never the control, so blocking at the network is the only lever. Decide that deliberately, not by accident.

Training versus search

Blocking a training crawler is a reasonable choice. Doing it by IP risks catching the search crawler from the same operator. robots.txt lets you split them cleanly, for example:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

The same logic applies to OpenAI; see whether to block GPTBot and still appear in ChatGPT search.

IP-only allowlisting

The reverse mistake also exists. OpenAI publishes its crawler IP ranges, but it also warns against relying on IP-only allowlisting. As of the evidence date, its published lists contain IPv4 prefixes only. Validate on user agent plus your CDN's verified-bot programme rather than a hand-maintained IP list.

1. Write your intent in robots.txt, per bot. Name the training, search and user agents you care about and give each an explicit rule. The per-bot robots.txt guide lists the tokens.
2. Remove IP blocks you didn't mean to keep. Check firewall rules, bot-management categories and any CDN defaults that changed on 2026-09-15.
3. Request /robots.txt as each crawler. Use each documented user agent and confirm a 200 with your real file in the body, not a 403, a 429 or a challenge page.
4. Request a real content page the same way. A reachable robots.txt with a blocked content URL is still a failure.
5. Retest after changes. Any firewall, CDN or hosting change can reopen the problem.

The step-by-step version is in making sure crawlers can fetch robots.txt itself. If you want a quick read on the network gate, 99Visibility's AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

None of this guarantees a citation. Passing both gates makes you eligible to be fetched. What an engine retrieves and cites after that is outside your control.

Is blocking IPs ever the right way to stop an AI crawler? It can be, for agents that don't follow robots.txt. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not apply to ChatGPT-User. For crawlers that do honour robots.txt, an IP block is blunter than it needs to be. It can't tell training from search, and it stops the crawler reading the rules you wrote. Use robots.txt first and reserve network blocks for cases where robots.txt isn't the control.

How do I know if my CDN is blocking AI crawlers? You test it from outside. Request your robots.txt and a content page using each AI crawler's documented user agent, and record the status code. A 200 with real content is a pass. A 403, a 429, or an HTML challenge or CAPTCHA page is a block. Then check your CDN's bot settings directly. Our guide to [CDN and firewall blocks on AI crawlers](/guides/cdn-firewall-blocking-ai-crawlers) covers where these settings usually live.

If robots.txt allows OAI-SearchBot, am I eligible for ChatGPT search? Only if the network also lets it through. OpenAI's help page says eligibility needs two things: allow OAI-SearchBot to crawl, and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Eligibility still isn't inclusion. OpenAI says ChatGPT ranks results using multiple factors and placement is not guaranteed.

Does allowing GPTBot help me appear in ChatGPT? No. OpenAI documents GPTBot as a training crawler and says each bot setting is independent of the others. ChatGPT search uses OAI-SearchBot. You can disallow GPTBot to opt out of training and still allow OAI-SearchBot for search, as long as your firewall lets OAI-SearchBot reach the site.

Can AI still mention my brand if its crawlers can't reach my site? Yes. A model can describe your brand from what it learned before its knowledge cutoff, or from other sites it can reach. Blocking access only removes your own pages as a source. That can mean the engine relies on third-party descriptions you don't control. See [whether AI can still talk about your brand after a block](/questions/block-ai-crawlers-brand-still-mentioned).