My robots.txt allows AI bots but I never show up in ChatGPT. What could block it?

A robots.txt allow tells a crawler it may fetch your pages. It does not prove the crawler got them, read them, or used them. For ChatGPT search, at least three things can still stop you after robots.txt says yes:

1. The wrong bot is allowed. Your rules name GPTBot, which governs training, while OAI-SearchBot, which governs search, is blocked or falls under a blanket rule.
2. The network says no. A CDN, firewall or bot-management rule returns a 403 or a challenge page to OpenAI's search crawler.
3. The content isn't in the HTML. Your prices, product names and claims load through JavaScript, and OpenAI's crawlers don't run it.

Then there is the part no one controls. ChatGPT decides per question whether to search at all. When it doesn't, the answer comes from the model's training data. If you clear every gate and still don't appear, the cause may sit there, not on your site.

Check the gates in the order above. Each one is cheap to test and each one fails silently.

OpenAI runs separate crawlers with separate jobs. Its crawler documentation says GPTBot collects content that may be used for training, and OAI-SearchBot surfaces sites in ChatGPT's search features. OpenAI states that "each setting is independent of the others." It also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

The common mistake: a site adds an explicit User-agent: GPTBot allow, keeps a User-agent: * disallow, and assumes ChatGPT is covered. OAI-SearchBot has no group of its own, so it follows the blanket rule and is blocked. The reverse mistake is a blanket AI block that removes search eligibility along with training.

If you want search without training, name both bots:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The reasoning behind that split is covered in whether to block GPTBot and still appear in ChatGPT search, and the per-bot robots.txt guide lists the equivalent tokens for Claude, Perplexity and others.

OpenAI's help center describes eligibility as a conjunction: allow OAI-SearchBot to crawl, and confirm that your host or CDN allows traffic from OpenAI's published search-bot IP addresses. robots.txt alone is not enough if the edge blocks the request.

This failure is invisible from your side. Your robots.txt says allow. Your WAF returns a 403, a rate limit or a JavaScript challenge. No dashboard warns you. OpenAI's own guidance notes that bot-protection systems can mistakenly block legitimate crawlers, often with 403 errors.

Things to look at:

• Bot-management and WAF rules that challenge or block unknown or "AI" traffic by category.
• CDN defaults that changed without you. Cloudflare began blocking Training and Agent crawlers by default on ad-displaying pages from 15 September 2026 for new domains, new sites and existing free-tier customers, with Search allowed by default. Read how that applies in whether Cloudflare's AI crawler settings could be hiding your site.
• IPv6-only origins. OpenAI's published IP files list IPv4 ranges only, so an AAAA-only edge can be unreachable.

The test is to request your pages with each crawler's user-agent and record the status code. The AI Crawler Checker does this and shows which crawlers your firewall turns away. For fixing what you find, see troubleshooting 403 errors for AI crawlers.

Vercel's crawler study (published December 2024, updated February 2025) found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. Independent re-testing in June 2026 found the same for GPTBot, OAI-SearchBot and ChatGPT-User.

Googlebot does render JavaScript. That's why a page can look perfect in Google and be close to empty for ChatGPT.

To check, fetch a key page with JavaScript disabled and compare the text with what a browser shows. If your pricing, product names or main claims appear only in the rendered version, OpenAI's crawlers never saw them. The fix is server-side rendering or prerendering for the content that matters. Content hidden behind tabs, accordions or "load more" buttons that fetch on click has the same problem.

Clearing access makes your page a candidate. It doesn't make ChatGPT pick it. OpenAI says ChatGPT ranks search results using multiple factors and that placement is not guaranteed. It also rewrites the user's prompt into its own search queries, which you can't see.

As of October 2026, three things sit outside your control:

• Whether search runs. ChatGPT searches at the model's discretion. Many answers use no live retrieval.
• What the model already knows. Without search, answers come from training data frozen at a cutoff. OpenAI documented a 16 February 2026 cutoff for the GPT-5.6 family and 31 August 2025 for the Instant path as of August 2026.
• Which retrieved sources get cited. A July 2026 critical survey of 45 GEO studies found no technique with a stable, cross-platform causal effect on discoverability.

What you can do beyond access: state specific facts (a price, a one-line definition) in fetchable HTML, and work on being named on the third-party sites ChatGPT already cites for your topic. That second route shows a strong association in the evidence, not a promised effect. If crawlers are visiting and you still aren't cited, why AI crawlers visit but never cite you covers the next layer.

Does allowing GPTBot help me appear in ChatGPT? No. GPTBot governs whether your content may be used to train OpenAI's models. OpenAI asserts no search benefit from it. OAI-SearchBot is the crawler tied to ChatGPT search answers. You can allow OAI-SearchBot and block GPTBot, and OpenAI documents that combination as valid. Blocking GPTBot also doesn't remove you from ChatGPT search.

Should I allowlist OpenAI's IP addresses in my firewall? OpenAI's help center says your host or CDN must allow traffic from its published search-bot IPs, listed at openai.com/searchbot.json. OpenAI also warns against relying on IP ranges alone. Verify by user-agent plus your CDN's verified-bot program where one exists, and re-check after any firewall change, since the lists are updated.

My logs show GPTBot but never OAI-SearchBot. Is something blocking it? Possibly. If OAI-SearchBot requests never reach your origin, your CDN may be rejecting them at the edge, which your origin logs won't record. Check CDN logs for blocked or challenged requests too. The detailed causes are covered in [why your logs show GPTBot but not OAI-SearchBot](/questions/logs-show-gptbot-not-oai-searchbot).

How long after fixing access will I show up in ChatGPT? Nobody publishes a timeline, and nobody can promise you'll show up at all. Fixing access makes your pages eligible for ChatGPT search. Whether they are retrieved and cited depends on each query and on OpenAI's ranking. Answers given without search won't reflect your fix until a future model is trained.