Free Check: Is Your robots.txt Blocking ChatGPT Search?

The check answers one question: does your robots.txt let OpenAI's search crawler fetch the pages you care about?

OpenAI runs separate crawlers with separate jobs. As of October 2026, its crawler documentation describes them like this:

• OAI-SearchBot is used to surface websites in ChatGPT's search features. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links."
• GPTBot crawls content that may be used to train OpenAI's foundation models. Disallowing it signals that your content should not be used for training.
• ChatGPT-User fetches pages when a user's action triggers it. OpenAI says robots.txt rules "may not apply" to it.

OpenAI also states that "each setting is independent of the others." Blocking GPTBot does not remove you from ChatGPT search, and allowing GPTBot does not help you get found there. If you are weighing the training question, see whether to block GPTBot and still appear in ChatGPT search. For the full crawler list, read OpenAI's crawler documentation, digested.

This takes about five minutes. You need a browser or a terminal.

1. Fetch the file. Open https://yourdomain.com/robots.txt in a browser, or run curl -s https://yourdomain.com/robots.txt. If it returns an error page or a firewall challenge, stop and fix that first (see whether AI crawlers can fetch your robots.txt).
2. List every User-agent line. Note whether OAI-SearchBot appears by name, whether GPTBot appears, and whether there is a User-agent: * group.
3. Find the group OAI-SearchBot obeys. Under the Robots Exclusion Protocol (RFC 9309), a crawler follows the group that names it. It only falls back to the * group when no group matches its name.
4. Read that group's rules. Disallow: / blocks the whole site. Disallow: /pricing blocks that path. When an Allow and a Disallow both match a URL, the more specific (longer) rule wins.
5. Test the pages that matter. Check your homepage, pricing, product and comparison pages against those rules. A site can be open overall and still block the one directory buyers ask about.
6. Repeat for subdomains. robots.txt applies per host, so shop.yourdomain.com and docs.yourdomain.com each need their own check.

Here is the blanket block that quietly removes ChatGPT search eligibility:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

Someone wanted to opt out of AI training and pasted a list that included search crawlers too. The other version is User-agent: * with Disallow: / and no named OAI-SearchBot group, which blocks every crawler that isn't named elsewhere.

Split training bots from search bots. If your goal is to keep content out of OpenAI's training while staying in ChatGPT search, this is the minimal pattern:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Then work through these steps:

• Remove OAI-SearchBot from any shared block list.
• Give OAI-SearchBot its own named group, so a strict * rule no longer applies to it.
• Keep any path you genuinely want private under Disallow in that group.
• Re-fetch the live file after deploying, since CDNs and CMS plugins sometimes serve a cached or generated version.

The same split applies to other engines. Anthropic, Perplexity and Meta each separate training crawlers from search crawlers, with their own tokens. A starting point for all of them is in the robots.txt template for AI search and training bots.

Your result falls into one of four combinations.

If OAI-SearchBot is blocked and nobody chose that, fix it now. If it is allowed and you still never appear, robots.txt is not your problem, and the next section explains where to look.

robots.txt is only the first gate. OpenAI's help center says eligibility needs two things: allow OAI-SearchBot to crawl, and confirm your host or CDN allows traffic from OpenAI's published search crawler IP addresses. A file that says allow while your firewall returns 403 leaves you just as invisible, and nothing in robots.txt shows it.

CDN defaults can also change without you editing anything. From 2026-09-15, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed by default. Read how Cloudflare's AI crawler settings could hide your site, then run the free firewall test for AI bots.

Three more limits:

• Eligibility is not citation. ChatGPT decides per query whether to search, and OpenAI discloses only that it ranks results using "multiple factors." Nobody controls that.
• User-triggered fetches differ. robots.txt may not apply to ChatGPT-User, so blocking it is not a reliable opt-out.
• Content after JavaScript. OpenAI's crawlers do not execute JavaScript, so text that only appears after scripts run may not be read even when access is allowed.

For a full walk-through of these causes, see why you might not show up in ChatGPT when robots.txt allows AI bots.

The robots.txt Checker for AI Crawlers on 99visibility.com fetches your live file and resolves the rules for each AI crawler by name, including OAI-SearchBot and GPTBot. It shows which group each bot follows and whether it is allowed, so you skip the manual parsing in steps 2 to 4.

Use it as a first pass, then confirm the firewall side separately. A clean robots.txt result tells you the stated rule. It does not tell you what your CDN actually returns to the crawler.

Does blocking GPTBot remove my site from ChatGPT search? No. OpenAI's crawler documentation says each setting is independent. GPTBot governs whether content may be used to train foundation models. OAI-SearchBot governs whether pages can appear in ChatGPT search answers. You can disallow GPTBot and allow OAI-SearchBot in the same file, and many sites do exactly that to opt out of training while staying eligible for search.

Will allowing GPTBot help ChatGPT find or cite my site? Nothing in OpenAI's documentation says so. GPTBot's description covers training only, and OpenAI asserts no search benefit from allowing it. Any effect on what a model knows would only appear in a future model generation, and nobody has measured it. For ChatGPT search, the crawler that matters is OAI-SearchBot, plus a firewall that lets it through.

My robots.txt has no mention of OAI-SearchBot. Am I blocked? It depends on your catch-all group. With no named group, OAI-SearchBot follows the rules under User-agent: *. If that group says Disallow: /, ChatGPT search is blocked. If the catch-all allows your pages, or there is no catch-all at all, OAI-SearchBot is permitted. Adding a named OAI-SearchBot group makes your intent explicit either way.

How quickly does a robots.txt fix take effect in ChatGPT? OpenAI does not publish a timeline. The crawler has to fetch your updated robots.txt and then recrawl your pages before anything can change. Even after that, appearing in an answer depends on whether ChatGPT searches for a given query and which results it selects. Fix the rule, confirm the live file, and check the firewall, then give it time.

Can my server logs confirm OAI-SearchBot is visiting? Yes, if you filter for its user-agent and check the status codes it receives. Requests returning 200 mean it reached your pages. Repeated 403s point to a firewall or bot-management rule. Because user-agent strings can be spoofed, OpenAI publishes its search crawler IP ranges at openai.com/searchbot.json so you can verify that a request is genuine.