IP allowlisting: definition and crawler caveats

IP allowlisting (older term: IP whitelisting) is a network rule that trusts traffic from a named set of IP addresses or ranges. Everything else either gets blocked or goes through normal bot screening.

In AI visibility work, the term usually means one thing: telling your firewall, CDN or web application firewall (WAF) to let a specific AI crawler through. The crawler's operator publishes the ranges it crawls from. You add them to an exception rule.

For the specific question of whether to do this for OpenAI, see whether to allowlist OpenAI's crawler IP addresses. This entry stays with the definition and the caveats.

Crawler operators publish machine-readable IP lists. OpenAI publishes one per bot as JSON files, including https://openai.com/searchbot.json for OAI-SearchBot. Each file lists IP prefixes and a creation time.

A typical setup looks like this:

1. Fetch the operator's published list.
2. Add those ranges to an allow or skip rule in your CDN or WAF.
3. Match the rule to the crawler's documented user-agent as well.
4. Refresh the list on a schedule, because operators update it.

Two details trip people up. First, when 99Visibility's evidence base checked OpenAI's files in August 2026, they contained IPv4 prefixes only and no IPv6 prefixes. That matters if your site resolves only over IPv6, which is covered in whether ChatGPT's crawlers can reach an IPv6-only site. Second, a list is a snapshot. OpenAI's chatgpt-user.json carried a mid-August 2026 creation date when checked, which shows these files are reissued.

Robots.txt is a request. The firewall is the gate. OpenAI's help centre states eligibility as a conjunction: allow OAI-SearchBot to crawl the site and confirm that the host or CDN allows traffic from OpenAI's published searchbot IP addresses.

If the WAF challenges or blocks the crawler, the page can drop out of ChatGPT search even though robots.txt says yes. OpenAI's advertiser guidance notes that bot-protection systems can mistakenly block legitimate crawlers, often returning 403 Forbidden errors. Nothing in your analytics flags it.

Blocking by IP has a second cost. Anthropic warns that IP blocking impedes its ability to read your robots.txt file, so your per-bot rules may never be seen. Details are in what happens when a blocked IP tries to read robots.txt.

Allowlisting only removes a block. It makes a page reachable, not more likely to be retrieved or cited. The next move is to test the gate directly: request a page as each AI crawler and compare the response with your robots.txt. The AI Crawler Checker does this and shows which crawlers the firewall turns away.

• "IP allowlisting is enough on its own." OpenAI publishes its IP ranges and still warns against IP-only allowlisting. Validate on user-agent plus a verified-bot method. Cloudflare's verified-bot criteria accept a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS. The steps are in verifying real AI crawlers versus spoofed user agents.
• "A user-agent match proves it's the real bot." Anyone can send any user-agent string. The IP or signature check is what ties the request to the operator.
• "Set it once and forget it." Operators reissue their lists. A hardcoded range from last year can silently exclude new crawl addresses.
• "If robots.txt allows the bot, the firewall must too." They are separate systems, often owned by separate teams. CDN defaults can also change without you editing anything: from 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains and existing free-tier customers.
• "Allowlisting gets you cited." It does not. It removes one block. Retrieval and citation are decided by the engine.

• OpenAI crawlers, robots.txt and firewalls: how GPTBot, OAI-SearchBot and ChatGPT-User differ, and where the firewall fits.
• Verified bot: a crawler your CDN has confirmed through an IP list, reverse DNS or a cryptographic signature.
• User-agent: the name a crawler sends with each request. Necessary for per-bot rules, not proof of identity.
• If robots.txt allows everything and you still never appear, start with what else can block you from ChatGPT.