Should I let AI crawlers access my website?

Most AI companies run more than one crawler, and each has a different job. One kind fetches pages for a search index that feeds live answers. Another collects content that may be used to train future models. A third fetches a page when a user asks the assistant to look something up.

OpenAI states the split plainly on its crawler page: "Each setting is independent of the others." You can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to opt out of training. OpenAI also says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links."

So the real question is two questions:

• Do you want to be eligible for AI search answers? Then allow the search crawlers.
• Do you want your content used for model training? That is a separate business and licensing call. It has no documented effect on search answers.

If you want the detail for each bot, the per-bot robots.txt guide covers the tokens and syntax.

Vendors document these roles on their own pages. They change, so recheck the source pages before you edit rules.

Two points catch people out. First, there is no robots.txt token that blocks Google AI Overviews or AI Mode. Google says Googlebot rules are the control, because AI is built into Search. Second, Anthropic runs ClaudeBot and Claude-SearchBot as separate agents, which is covered in whether to allow Claude-SearchBot if you block ClaudeBot.

Work through these steps in order.

1. Decide on AI search. If being found in ChatGPT, Perplexity or Claude search matters to you, allow OAI-SearchBot, PerplexityBot and Claude-SearchBot. Most businesses that sell anything will want this.
2. Decide on training separately. If you object to training use, disallow GPTBot, ClaudeBot and Google-Extended. Nothing in the evidence shows this costs you search eligibility. Nothing shows that allowing them helps you get cited either. The effect on what models remember about you is unmeasured by anyone. The training crawler definition explains what blocking does and does not do.
3. Leave Googlebot alone unless you want out of Google entirely. Blocking it removes you from Search and its AI features together.
4. Handle unknown bots on their merits. Check whether a crawler publishes its purpose before you block it. See whether to block AI crawlers you don't recognize.

A hypothetical: say you run a B2B software company and want AI search visibility but no training use. Your robots.txt could include:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The playbook for blocking AI training while staying in AI search walks through this setup step by step.

Allowing a search crawler makes you eligible. It does not make you retrieved, ranked or cited. Retrieval happens per query, at each engine's discretion, and no publisher controls it.

OpenAI's help center adds a second condition. To be eligible, you must allow OAI-SearchBot and "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." That is an "and". A firewall that returns 403 to the bot removes you as fully as a robots.txt block, and nothing in your dashboards will flag it.

This risk grew recently. From 2026-09-15, Cloudflare blocks Training and Agent crawlers by default on pages that display ads, for new domains and existing free-tier customers, while Search stays allowed by default. A site can change its AI access without anyone editing a file. The guide to CDN and firewall blocks on AI crawlers covers how to find these.

One more gate: Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. If your prices or product names only appear after JavaScript runs, an allowed bot may still see an empty page.

Treat this as a short audit, not a one-off edit.

• Read your robots.txt and list the rule that applies to each named AI crawler. Look for blanket AI blocks that also catch search bots. That is the most common mistake.
• Test the network layer. Request a page as each crawler and check the status code. The AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.
• Check your CDN bot settings, especially if you use Cloudflare and show ads.
• Check what the bot sees. Confirm key facts are in the raw HTML before JavaScript runs.

Once access is clean, the free checker on this site shows whether AI engines actually name you for three buyer questions. It is a snapshot, so rerun it after you change anything.