Should I allowlist OpenAI's crawler IP addresses in my firewall?

OpenAI's help article on ChatGPT search states the requirement plainly: "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The key word is and. robots.txt alone is not enough if a firewall sits in front of it.

So the answer to "should I allowlist?" is yes in outcome. Your edge has to let OAI-SearchBot's real traffic through. The question is how you write that rule.

OpenAI publishes its IP ranges and, on the same documentation, warns against IP-only allowlisting. That points to a rule built on two checks:

• The request carries the documented OAI-SearchBot user agent.
• The request is verified as genuine, through your CDN's verified-bot programme or a check against the published ranges.

Then you test it, because a rule that looks right in a dashboard can still return 403. Our breakdown of OpenAI's help article on searching the web with ChatGPT covers the surrounding guidance.

OpenAI runs separate crawlers, and each one has a separate consequence. Its bots page says "Each setting is independent of the others."

This matters for your firewall. Allowing GPTBot does nothing for search visibility, and blocking it does not remove you from ChatGPT search. If you are weighing that trade-off, see whether to block GPTBot and still appear in ChatGPT search.

OpenAI publishes IP ranges as JSON files, including one for OAI-SearchBot at openai.com/searchbot.json. When our evidence base checked them in August 2026, the files listed IPv4 prefixes only, with no IPv6 prefixes. OpenAI's own guidance also notes that bot-mitigation systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." For the full crawler list, read OpenAI's crawler documentation, digested.

Start with robots.txt, since it is the first gate. A minimal rule that allows search while keeping training separate looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The second block is optional. Keep or drop it based on your training preference.

Then set the edge rule. Each check alone has a weakness:

• User agent only: anyone can send the string "OAI-SearchBot". A rule that trusts it lets spoofed traffic in.
• IP only: you take on maintenance of a list OpenAI regenerates, and the rule says nothing about which bot is calling.

Combine them. Most CDNs run a verified-bot programme that does the IP or signature check for you. Cloudflare's criteria for a verified bot include "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS", plus robots.txt compliance. Allow verified OAI-SearchBot traffic, and keep challenges in place for unverified requests that only claim the name. Our playbook on verifying real AI crawlers versus spoofed user agents walks through the checks.

Several failures look like a correct setup from the outside.

• CDN defaults changed without you. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts and all existing free-tier customers. Search stays allowed by default. Check which category your zone assigns each OpenAI bot, as of October 2026.
• Challenges and CAPTCHAs. OpenAI notes that because its crawlers are automated, "these checks may block access." A JS challenge page counts as a failure even with a 200 status. See whether bot challenges block AI crawlers.
• IPv6-only origins. If the published ranges are IPv4 only, an AAAA-only path is unreachable regardless of robots.txt. Read whether ChatGPT's crawlers reach IPv6-only sites.
• robots.txt itself blocked. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." The same risk applies to any crawler your firewall turns away.

The wider pattern is in when your CDN or firewall blocks AI crawlers.

A rule is only proven when a request gets through. Run this check after any firewall change:

1. Request a representative content URL with the documented OAI-SearchBot user agent.
2. Record the status: 200 passes, 403 means bot mitigation, 429 means rate limiting.
3. Read the body. A challenge or CAPTCHA page is a failure.
4. Confirm /robots.txt is also reachable to the same request.
5. Repeat for the other bots you care about, such as PerplexityBot and Claude-SearchBot.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. For a manual method, follow testing whether your firewall lets AI crawlers in.

Be clear on what a pass buys. Allowing the bot gives you eligibility for ChatGPT search. It does not control retrieval or citation, which the engine decides per query. What you control is removing the block.