SEO for AI Search Engines: A Per-Engine Map of Crawlers, Indexes and Controls

Every AI answer passes through up to four stages, and only one of them is fully yours.

• Parametric: what the model learned in training, frozen at a knowledge cutoff. Nothing you publish this quarter changes it.
• Access: whether a crawler may fetch and store your page. You and your CDN control this.
• Retrieval: whether your page enters the model's context for a given question. The engine's operator controls most of it.
• Generation: which retrieved pages the model uses and cites. The model decides.

Retrieval is also discretionary on every platform. Google AI Overviews often do not trigger. ChatGPT decides per question whether to search. Claude.ai search is a user toggle. So an answer about your brand may come from live pages or from memory, and you usually cannot tell which.

What this means in practice: SEO for AI search engines starts at the access stage, and the access stage is different for every engine. The map below is the starting point.

Each row names the crawler that feeds the engine's search features, the index it draws on and the control the operator documents. Vendors add and rename bots, so check each operator's page before you edit rules.

Two patterns stand out. Most vendors split training, search and user-triggered fetching into separate bots. And Google is the exception: its AI search features ride on Googlebot and the normal Search index, so you cannot block them with a separate token. The difference between Google-Extended and Googlebot is the most misread line in this table.

Many sites use a plugin or CDN setting that blocks "AI bots" as one group. That bundles decisions that each vendor documents separately.

Say you run a software company and want to keep your content out of model training. A blanket rule that disallows GPTBot and OAI-SearchBot together does that, and it also removes you from ChatGPT search. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. OpenAI also states that each setting is independent, so you can disallow GPTBot and still allow OAI-SearchBot.

A blanket allow fails the other way. Allowing GPTBot does not help you appear in ChatGPT search, because GPTBot governs training only. A short per-bot file makes the intent explicit:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Indexes diverge too. In OtterlyAI's 2026 experiment with two sites of AI-generated posts, Google deindexed both sites and its AI surfaces almost never cited them. Bing kept both sites, and Microsoft Copilot accounted for 2,718 of the 3,934 citations Otterly tracked for one of them. ChatGPT cited them effectively zero times even though they ranked in Bing. One engine's index told Otterly nothing about another engine's answers.

The next move: decide training and search access per vendor, then write rules per bot. A robots.txt template for AI search and training bots gives you a starting file. For why tactics rarely transfer cleanly, see whether what works on one AI engine works on the others.

In your control

• Per-bot robots.txt permission for each search crawler in the map.
• Network reachability. OpenAI's help centre says eligibility requires allowing OAI-SearchBot and confirming your host or CDN allows OpenAI's searchbot IP addresses. Both halves matter.
• Server-side rendering of key facts. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. Googlebot renders. If your pricing appears only after JavaScript runs, most AI crawlers never see it.
• Snippet controls. noindex, nosnippet, max-snippet and data-nosnippet keep content out of AI answers on Google and Bing surfaces. Check that nobody applied them by accident.
• Stating specific facts (a price, a number, a one-line definition) on a page crawlers can fetch, so an engine has a sentence it can quote.
• Being named on third-party pages the engines already cite. The evidence shows an association here, not a promised effect. Start by finding the pages AI engines cite in your category.

Not in your control

• The queries engines actually run. ChatGPT rewrites prompts, and Google can fan a question out into several searches.
• Where your page lands among retrieved sources, and whether a retrieved page gets cited.
• Whether an AI Overview triggers for a query at all.
• What a model learned before its cutoff.

Google states there are no additional requirements or special optimizations needed to appear in AI Overviews or AI Mode beyond being indexed and snippet-eligible. That rules out a separate markup path into Google's AI features.

Work from the first gate outward. Each step depends on the one before it.

1. Write the policy per vendor. For OpenAI, Google, Anthropic, Perplexity, Microsoft and Meta, decide two things: training yes or no, search yes or no.
2. Encode it per bot. Edit robots.txt on every host that serves content. Rules do not carry across hosts, so check whether each subdomain needs its own robots.txt rules.
3. Test from outside as each bot. Request real pages with each crawler's user agent and record the status code. A 403 or a challenge page means the firewall overrides your robots.txt. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away. For a manual method, follow how to test robots.txt rules for each AI bot.
4. Review CDN defaults. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stays allowed. You can lose access without editing anything.
5. Diff raw HTML against the rendered page. Move prices, product names and key claims into the server response.
6. Audit snippet directives and Search Console. Look for inherited nosnippet or data-nosnippet, and check whether the generative AI control is set on your property or a parent.
7. Check index status per engine. Use Google Search Console and Bing Webmaster Tools separately. Healthy in one says nothing about the other.
8. Measure per engine. A single blended score hides engine-level gaps. The free checker on this site gives you a snapshot for up to three buyer questions, scored per engine.

• One switch for all AI bots. It removes search eligibility you probably wanted to keep.
• Allowing GPTBot to get into ChatGPT. GPTBot is training only. OAI-SearchBot is the search control.
• Trusting robots.txt alone. A WAF or bot-management rule can return 403 to a crawler your file allows, and nothing in your dashboards warns you.
• Expecting robots.txt to stop user-triggered fetchers. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
• Using Google-Extended to hide from AI Overviews. It controls Gemini Apps and Vertex AI training and grounding, not Search features.
• Adding llms.txt, FAQ schema or AI-friendly rewrites as the fix. These are not access levers, and the claims that they get you cited lack credible evidence. Google says Search ignores llms.txt.
• Mass-publishing thin pages. Otterly's AI-generated sites were removed from Google with no manual action shown in Search Console.
• Hidden text or cloaking for AI crawlers. Research shows injected content can manipulate search agents once retrieved. It is illegitimate, it targets the engines' users, and it puts the whole domain at risk. Do not use it.

Does blocking GPTBot remove my site from ChatGPT search? No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, and states that each setting is independent. You can disallow GPTBot to keep content out of training while allowing OAI-SearchBot so your pages stay eligible for ChatGPT search answers. Allowing OAI-SearchBot buys eligibility only. It does not guarantee retrieval or citation.

Can I keep content out of Google AI Overviews but stay in Google Search? Not with robots.txt, because Google's AI features use Googlebot and there is no separate token. The documented controls are nosnippet, max-snippet and data-nosnippet, which limit what Google can show, including in AI features. Google also offers a Search Console control for generative AI features. It is opt-out, inclusion is the default, and as of October 2026 it has reached only a subset of properties.

Which AI search engines list should I allow in robots.txt? Start with the search crawlers: OAI-SearchBot, Googlebot, bingbot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer. Then decide separately on training crawlers such as GPTBot, ClaudeBot and Google-Extended. Keep the list current, since vendors add bots. After editing, test each one against your live firewall, because an allow rule means nothing if the CDN returns 403.

Do schema markup or llms.txt help AI search engines cite me? The evidence does not support that. Google says there are no special optimizations needed for AI Overviews or AI Mode, and that Search ignores llms.txt. No other engine documents reading llms.txt. Existing schema does no harm, but it is not a lever for AI answers. Your effort is better spent on crawler access, server-rendered facts and snippet settings.

Why does my site appear in Copilot but not in ChatGPT? They do not share a single index or retrieval process. Copilot grounds on Bing, while ChatGPT search uses OpenAI's own crawl and sometimes other providers. In Otterly's 2026 test, sites that ranked in Bing were cited heavily by Copilot and effectively never by ChatGPT. Check OAI-SearchBot access and firewall reachability first, then measure each engine on its own.