OpenAI's Crawler Documentation, Digested: GPTBot, OAI-SearchBot and ChatGPT-User

The source is OpenAI's own page, "Overview of OpenAI Crawlers", on its developer site. It is the first-party description of the user agents OpenAI sends to websites and what each one is for.

The page is undated. Our evidence base fetched and verified it on 2026-08-22, and the quotes below are taken from that verified copy. OpenAI can change the page without notice, so treat this digest as accurate as of October 2026 and check the live page before you change production rules.

The page matters because most robots.txt advice about ChatGPT gets the agents mixed up. OpenAI separates training, search and user-triggered fetching into three agents. Each has a different consequence when you block it. If you want the wider picture of access, firewalls and logs, see OpenAI crawlers, robots.txt and firewalls.

We did not test OpenAI's crawlers. This page reports what OpenAI says, read against the rest of our verified evidence base.

• Primary text first. Every claim below comes from a fetched copy of OpenAI's bots page or its help article on ChatGPT search, not from a summary.
• Adversarial review. In the evidence base, a second agent re-checked each claim against the source, looking for over-reading.
• Scope. OpenAI's statements are assertions about its own systems. They are the best available evidence for how OpenAI's agents behave, but they are not a measured study of outcomes.

That last point sets the limit of this digest. The document tells you what each control does. It does not tell you how often a page gets cited once it is allowed.

GPTBot: training

OpenAI writes: "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."

Nothing in GPTBot's entry mentions search, retrieval or citation.

OAI-SearchBot: ChatGPT search

OpenAI writes: "OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

ChatGPT-User: user-initiated fetches

OpenAI says ChatGPT-User is "not used to determine whether content may appear in Search" and adds: "Because these actions are initiated by a user, robots.txt rules may not apply."

The settings are independent

OpenAI states: "Each setting is independent of the others." Its example: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."

The second gate: network access

OpenAI's help article on ChatGPT search adds a condition the bots page does not spell out: "To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The word is "and". A robots.txt allow does nothing if your firewall returns a 403.

Error 1: "Allow GPTBot so ChatGPT can cite you"

GPTBot's documented purpose is training. OpenAI asserts no retrieval or citation benefit from allowing it anywhere on the page. Our evidence base labels any effect of training inclusion on what a model later says about you as unmeasured, not proven and not disproven. Any effect would also arrive a full model generation later, bounded by each model's knowledge cutoff. If a vendor tells you GPTBot access gets you cited, the source they need to cite does not say so.

Error 2: "Blocking GPTBot hides you from ChatGPT search"

OpenAI's independence statement contradicts this directly. Search inclusion runs through OAI-SearchBot. You can disallow GPTBot and allow OAI-SearchBot, and OpenAI gives exactly that as its example. The real-world mistake runs the other way: a blanket block on every AI agent quietly removes OAI-SearchBot too, and with it ChatGPT search eligibility. Our answer to whether to block GPTBot if you still want ChatGPT search walks through the trade-off.

• No ranking or placement detail. OpenAI's whole disclosure on ranking is: "ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information. Placement is not guaranteed." Allowing OAI-SearchBot makes you eligible. It does not make you retrieved or cited.
• No guarantee search runs at all. ChatGPT searches at the model's discretion, and OpenAI says it "typically rewrites your query into one or more targeted queries". Many answers come from the model's training data, where your robots.txt today has no effect.
• No rendering promise. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but "don't execute them". Content that appears only after JavaScript runs may be invisible to OAI-SearchBot even when access is allowed.
• No user-fetch control. Because robots.txt may not apply to ChatGPT-User, a disallow is not a reliable way to stop user-triggered fetches.
• No frequency data. The document explains mechanisms. It does not measure how often sites block the wrong agent or how often firewalls override robots.txt.

Other vendors split their agents differently, so do not copy OpenAI's logic across. Compare Anthropic's crawler documentation before writing one shared rule.

Decide per agent, based on what you want.

1. Want ChatGPT search eligibility? Allow OAI-SearchBot. This is the only OpenAI agent tied to search inclusion.
2. Want to keep content out of training? Disallow GPTBot. That does not affect search eligibility.
3. ChatGPT-User: a rule here is a signal at best. OpenAI says robots.txt may not apply to user-initiated fetches.

A minimal pattern for "search yes, training no":

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Then check the parts robots.txt cannot cover:

• Firewall and CDN. Confirm your host lets OpenAI's published search IP addresses through. OpenAI also warns against IP-only allowlisting, so verify on user agent plus your CDN's verified-bot settings. Cloudflare's new defaults from 2026-09-15 block Training and Agent categories on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed by default. Review your dashboard rather than assuming.
• Logs. If you see GPTBot but never OAI-SearchBot, read why logs show GPTBot but not OAI-SearchBot.
• Rendering. Make sure prices, product names and key facts are in the HTML before JavaScript runs.

For a full file covering other vendors, start from our robots.txt template for AI bots. To see whether your firewall turns OAI-SearchBot away despite an allow rule, the AI Crawler Checker requests a page as each AI crawler and shows which ones get blocked.

Is OAI-SearchBot the same as the bot that answers when a user pastes a link? No. OpenAI documents ChatGPT-User as the agent for actions a user initiates, such as asking ChatGPT to open a page. OpenAI says it is not used to decide whether content may appear in search, and that robots.txt rules may not apply to it. OAI-SearchBot is the separate agent that surfaces sites in ChatGPT search results.

If I opt out of OAI-SearchBot, will ChatGPT never link to me? Not quite. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Opting out removes you from search answers, not from every possible link. If you want to be eligible for ChatGPT search answers, allow OAI-SearchBot and make sure your firewall lets it through.

My robots.txt allows OAI-SearchBot. Why might I still be missing from ChatGPT? Eligibility has two parts in OpenAI's own wording: robots.txt permission and network access from OpenAI's published search IP addresses. A firewall or CDN can return a 403 regardless of robots.txt. Content rendered only by JavaScript may also be invisible. Even with both gates passed, OpenAI says placement is not guaranteed. See [what else can block ChatGPT](/questions/robots-allows-ai-bots-not-in-chatgpt).

Does blocking GPTBot make ChatGPT forget my brand? Nothing published measures that. GPTBot governs whether content may be used in future training, so any effect would land in a later model generation, not in today's answers. Current models already have fixed knowledge cutoffs. OpenAI makes no claim either way about brand recall, so treat the effect as unmeasured and decide on training use as a content-rights question.

How do I quickly check whether my robots.txt blocks ChatGPT search? Look for a User-agent: OAI-SearchBot group with Disallow rules, and for a wildcard group that disallows everything without a specific OAI-SearchBot allow. Blanket AI blocks are the common culprit. Our [free ChatGPT search block check](/free/free-chatgpt-search-block-check) reads your file for you. Remember that a clean robots.txt does not rule out a firewall block.