OpenAI Crawler User Agents: robots.txt, IP Ranges and Firewall 403s

OpenAI documents three agents on its bots page. They do different jobs, and robots.txt treats each one separately.

The most common real-world error is a blanket AI block. A site owner wants to keep content out of training, blocks every OpenAI agent, and silently removes ChatGPT search eligibility at the same time. If you want the full vendor wording, see our digest of OpenAI's crawler documentation.

OpenAI's bots page says it plainly: "Each setting is independent of the others." Its own example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training.

That means two myths fall at once. Allowing GPTBot does not help ChatGPT find you. Blocking GPTBot does not remove you from ChatGPT search.

A robots.txt that blocks training and allows search looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Write one group per agent you care about. Then check your wildcard group too, because a User-agent: * disallow can catch agents you never named. Our robots.txt template for AI search and training bots covers the other vendors' agents in the same pattern.

ChatGPT-User is the odd one out. OpenAI describes it as acting on behalf of a user, and states: "Because these actions are initiated by a user, robots.txt rules may not apply."

What that means in practice:

• A disallow for ChatGPT-User is not a reliable way to stop ChatGPT from fetching a page a user asked about.
• Blocking ChatGPT-User does not affect search inclusion, because OpenAI says it is not used to determine whether content may appear in Search.
• Index-time control (OAI-SearchBot) and answer-time fetching (ChatGPT-User) are different controls. Treat them separately.

If you genuinely need content kept out of every fetch, robots.txt is the wrong tool. Access controls on the server, such as authentication, are what actually stop a request.

robots.txt is only half of eligibility. OpenAI's ChatGPT search help article says to "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The word that matters is "and".

OpenAI publishes its ranges as JSON files, including https://openai.com/searchbot.json. When the evidence base checked them in August 2026, searchbot.json listed 35 IPv4 prefixes and no IPv6 prefixes. The GPTBot and ChatGPT-User files were also IPv4-only. If your bot-facing edge only answers over IPv6, see whether ChatGPT's crawlers can reach an IPv6-only site.

Bot protection is where most silent failures start. OpenAI's own guidance warns that these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors," and that JavaScript challenges or CAPTCHAs may block automated crawlers. Your robots.txt says allow. Your firewall says 403. No dashboard tells you.

One dated change matters as of October 2026. From 15 September 2026, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains, new sites on existing accounts, and existing free-tier customers, while Search stays allowed by default. A site can change its AI access without its owner editing anything, so check your bot settings even if you have not touched them. Our comparison of robots.txt vs firewall rules explains how the two gates interact.

You can run a first-pass test in a few minutes. It tells you whether rules keyed on the user agent are turning OpenAI away.

1. Copy the full user-agent string for OAI-SearchBot from OpenAI's bots page. Do not shorten it.
2. Request a page that matters, such as your pricing page, with that user agent:

curl -I -A "<full OAI-SearchBot user-agent string>" https://example.com/pricing

3. Read the status code. A 200 passes. A 403 is a bot-mitigation block. A 429 is rate limiting. A 200 whose HTML is a challenge or CAPTCHA page is also a failure.
4. Request /robots.txt the same way. If the crawler cannot read robots.txt, it cannot read your permissions either.
5. Repeat for GPTBot and ChatGPT-User, and from both a datacenter and a residential connection, because some rules key on the network.

Know the limit of this test. Your request comes from your IP, not OpenAI's. A firewall that verifies bots by IP should block your spoofed request, so a 403 here may be correct behaviour. The ground truth is your origin or CDN logs filtered to verified OpenAI traffic, which show what the real crawler received. Our firewall testing playbook walks through both checks. The AI Crawler Checker also requests a page as each AI crawler and shows which ones the firewall turns away.

The tempting response is to paste OpenAI's ranges into an allowlist and move on. OpenAI publishes the IP files and also warns against relying on IP-only allowlisting. The better pattern is to validate on user agent plus your CDN's verified-bot programme, so legitimate crawlers get through and spoofed ones do not. Read more in our explainer on verifying real AI crawlers vs spoofed user agents.

What passing both gates gets you

Allowing OAI-SearchBot and clearing the firewall buys eligibility. It does not buy retrieval or ranking. OpenAI's ranking disclosure says ChatGPT ranks results "using multiple factors" and that "placement is not guaranteed."

One more access issue sits behind the firewall. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. If your prices or product facts only appear after JavaScript runs, OAI-SearchBot may receive a page without them. Check that the facts you care about are in the raw HTML.