What is an AI crawler?

An AI crawler is a bot that requests pages on behalf of an AI company. It identifies itself with a user-agent string, such as GPTBot, ClaudeBot or PerplexityBot. People also call them AI bots or LLM crawlers.

They differ from classic search bots in purpose, not mechanics. Both request HTML over HTTP. What changes is what the company does with the page afterwards. For a deeper comparison, see SEO bots vs AI crawlers.

Operators now split their crawling into three jobs, each with its own bot.

• Training crawlers collect pages that may be used to train foundation models. Examples: GPTBot, ClaudeBot, Meta-ExternalAgent. Google uses the Google-Extended token to control Gemini training and grounding, and says it is not a Search ranking signal.
• Search indexing crawlers build the index an AI search feature retrieves from. Examples: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Meta-WebIndexer.
• User-triggered fetchers load a page when a person asks the assistant to. Examples: ChatGPT-User, Claude-User, Perplexity-User. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules."

Most of these bots read raw HTML only. Vercel's 2024 crawler study found ChatGPT and Claude crawlers fetched JavaScript files (11.50% and 23.84% of requests) but didn't execute them. Independent re-tests as of June 2026 found the same for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider. Googlebot and AppleBot render JavaScript.

Crawler access is the first gate. If a search crawler can't fetch your page, the engine has nothing of yours to retrieve. This is one of the few areas where the evidence is strong and you hold the controls.

Three things can block you:

1. robots.txt rules. These are per bot. A blanket AI block often removes ChatGPT search eligibility by accident. Our per-bot robots.txt guide covers each token.
2. Your CDN or firewall. OpenAI's help centre says eligibility needs both permission for OAI-SearchBot and a host or CDN that lets its published IP addresses through. As of September 15, 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and free-tier customers. See whether Cloudflare could be hiding your site.
3. JavaScript rendering. Prices, product names and key claims that load only after scripts run may be invisible to non-Google AI crawlers. Server-side render the content that matters.

The next move: request your page as each crawler and compare the result with your robots.txt. The free AI crawler checker does this. Access buys eligibility only. It does not guarantee retrieval or citation.

• "Allow GPTBot so ChatGPT can find you." GPTBot governs training only. ChatGPT search uses OAI-SearchBot. OpenAI states "Each setting is independent of the others."
• "Blocking GPTBot removes you from ChatGPT." It doesn't. You can block training and still allow search.
• "robots.txt is decisive." It's necessary, not sufficient. User-triggered fetchers may skip it, and a firewall can block a bot robots.txt allows. If you're allowed but absent, read what else could block ChatGPT.
• "Crawler visits mean visibility." In Otterly's 2026 experiment, ChatGPT's crawlers kept visiting two test sites that earned effectively zero ChatGPT citations.
• "There's a bot to block for AI Overviews." There's no robots.txt token for AI Overviews or AI Mode. Googlebot rules control all Google surfaces.

• OpenAI crawlers, robots.txt and firewalls: GPTBot, OAI-SearchBot and ChatGPT-User in detail.
• CDN and firewall blocking: how a 403 silently removes you.
• robots.txt: the per-bot permission file defined in RFC 9309.
• Server-side rendering: putting content in the initial HTML so non-rendering bots can read it.

Which AI crawlers can render JavaScript? Among tested crawlers, only Googlebot and AppleBot render JavaScript. Vercel's study and later re-tests found that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider don't execute it. If your key facts load client-side, check the raw HTML before scripts run. Anything missing there may be invisible to ChatGPT, Claude and Perplexity crawlers.

Should I block training crawlers but allow search crawlers? That's a valid choice, and operators support it. OpenAI lets you disallow GPTBot while allowing OAI-SearchBot. Blocking training bots keeps your content out of future model training. Nothing published measures whether that affects how models describe you later. Allowing search bots keeps you eligible for AI search answers, though it guarantees nothing about citation.

Can I verify that a bot is really an AI crawler? Partly. OpenAI publishes IP ranges for its bots, including OAI-SearchBot, but warns against relying on IP allowlisting alone. Validate on user-agent plus the operator's verified-bot programme or published IPs. Cloudflare's verified-bot criteria include a cryptographic signature, a published IP list with a stable user-agent, or reverse DNS. Your server logs show which bots actually arrive.

Does blocking AI crawlers hurt my Google rankings? Google says the Google-Extended token, which controls Gemini training and grounding, does not affect inclusion in Google Search and is not a ranking signal. Blocking Googlebot is different: it controls Search and all Google AI surfaces together. Treat each bot as its own decision and check what each operator documents.