Search crawler: what it is and what it does in AI search

A search crawler is an automated web crawler that an AI platform runs to fetch and store pages for its search index. When a user asks a question and the engine decides to search, it draws candidates from that index.

Three kinds of AI bots get confused with each other:

• Search crawlers build the search index. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer are examples.
• Training crawlers collect content that may train future models. GPTBot and ClaudeBot are examples.
• User-triggered fetchers load a page when a person asks the assistant to. ChatGPT-User and Perplexity-User are examples.

Each one has its own user-agent token, and each answers to its own robots.txt rule.

The crawler requests your page, reads the HTML it gets back, and stores what it can extract. That stored copy is what the engine can later pull into retrieval for a specific question.

Two details decide what actually gets stored:

• Permission. The crawler checks robots.txt for rules addressed to its token. OpenAI's crawler documentation states that "each setting is independent of the others", so you can allow OAI-SearchBot while disallowing GPTBot.
• Rendering. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Prices, product names or claims that only appear after JavaScript runs may be missing from the stored copy. Googlebot is the main exception, because it renders.

A minimal rule that allows ChatGPT search while opting out of training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

A search crawler is the first gate. If it can't fetch your page, that engine's search answers have nothing of yours to retrieve. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links."

The gate has two halves. OpenAI's help center says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses. A firewall that returns 403 removes you as quietly as a Disallow line does. The two gates for AI crawlers are worth checking together.

What to do next:

1. Read your robots.txt rule for each search crawler token, not just the wildcard.
2. Request a page as each crawler and confirm you get a 200, not a challenge or a 403. The AI Crawler Checker shows which crawlers your firewall turns away.
3. Confirm your key facts are in the raw HTML before JavaScript runs.

Google is different. There is no separate robots.txt token for AI Overviews or AI Mode. Google says a page must be indexed and eligible for a snippet, and that there are no additional technical requirements.

"Allowing the crawler gets me cited"

It gets you eligible. As of October 2026, retrieval is per query and discretionary on every major platform, and the model chooses which retrieved pages to cite. Meta is the one vendor that says allowing its index crawler "helps us cite and link to your content." No vendor promises an outcome.

"Allow GPTBot so ChatGPT can find me"

GPTBot governs training only. ChatGPT search uses OAI-SearchBot. See whether allowing search bots means training use for the full split.

"A blanket AI block only stops training"

A wildcard block of every AI bot also removes search eligibility. Here is what blocking all AI bots does.

"robots.txt stops every AI visit"

User-triggered fetchers follow different rules. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules.

• Retrieval: the answer-time step that pulls indexed pages into the model's context.
• Agentic search: assistants that browse and fetch pages on a user's behalf, often through user-triggered fetchers.
• Claude-SearchBot: Anthropic's search crawler, distinct from ClaudeBot.
• robots.txt template for AI bots: per-token rules for search and training crawlers.