SEO Bots vs AI Crawlers: Two Different Things, and Why It Matters

People who search for "seo bot ai" usually mean one of two very different things.

• AI SEO chatbots and tools. Software you log into. It drafts articles, suggests keywords, rewrites titles or audits pages. It works for you, the publisher.
• AI crawlers. Automated clients run by OpenAI, Anthropic, Perplexity, Google, Meta and Microsoft. They request your pages over HTTP. They work for the AI platform.

The first category has no direct line into ChatGPT, Perplexity or Google AI Overviews. Nothing you type into an SEO chatbot reaches those engines. The second category is the only route your content takes into an AI answer that uses live retrieval.

The tools carry their own risk when used for volume. OtterlyAI published a 2026 test of two new sites loaded with about 1,000 AI-generated posts each: Google deindexed one after about 14 days and the other after about 90, with no manual action reported. Google's policy targets many pages made to manipulate rankings without adding value, not AI text as such.

The rest of this page is about the crawlers, because that's where the controls are.

Each AI company runs several bots with different jobs. The job decides what blocking it does. This table lists the documented consequences as of October 2026, from each operator's own crawler pages.

Two details trip people up. There is no robots.txt token that targets AI Overviews or AI Mode alone: Googlebot is the control there. And Google-Extended is a permission token, not a separate visitor you'll see in logs.

For the per-bot rules in detail, see the per-bot robots.txt guide. For the user-fetch group, see the guide on user-triggered AI fetchers.

An AI answer that uses your page passes through four stages. Crawlers only touch one of them.

1. Parametric. What the model knows without web access, frozen at a training cutoff. Training crawlers feed future models, but nothing published measures whether allowing them changes what a model says about you.
2. Access. Whether a crawler may fetch and store the page. This is the crawler stage, and it's the one you control.
3. Retrieval. Whether the page is pulled into the model's context for a given question. The platform decides.
4. Generation. Which retrieved pages get used and cited. The model decides.

Access has several gates, and a page has to pass all of them:

• robots.txt must allow the specific bot.
• The firewall or CDN must let the request through. OpenAI's help centre says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses.
• The HTML must contain the content. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Googlebot and AppleBot are the documented exceptions.
• Snippet and index controls such as noindex, nosnippet, max-snippet:0 and data-nosnippet must not suppress the content.

The two gates of robots.txt and firewall cause the most silent failures, because they can disagree without anyone noticing.

In your control

• Which bots your robots.txt allows, one agent at a time.
• Whether your firewall, CDN or bot-management rules return a 200 or a 403 to each crawler.
• Whether prices, product names and key facts are in the server-rendered HTML before JavaScript runs.
• Whether templates carry accidental noindex or nosnippet directives.
• Your CDN settings. Since 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can lose access without anyone editing it.

Not in your control

• Whether a platform searches at all for a given question. ChatGPT search is model-discretionary, and Claude.ai search is a user toggle.
• Which retrieved pages get cited.
• What a model learned before its cutoff.
• Whether user-triggered fetchers honour robots.txt. Their operators say they may not.

The honest summary: allowing a crawler makes you eligible. It does not make you retrieved, ranked or cited. The levers that hold up under scrutiny work by removing blocks, and none of them add a bonus.

Work from the outermost gate inward. Each step either finds a block or rules one out.

1. Decide your policy per purpose. Many teams want search visibility but no training use. That is a legitimate split, because OpenAI states each setting is independent.
2. Write robots.txt per bot. A short example that allows ChatGPT search while opting out of OpenAI training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

3. Test the firewall. Request a real content URL with each crawler's user agent and record the status code. A 403, 429 or CAPTCHA page means the bot is turned away whatever robots.txt says. The firewall testing playbook walks through it, and the AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.
4. Check rendering. Compare the raw HTML with the rendered page. Anything that appears only after JavaScript runs is likely invisible to non-Google AI crawlers. See the list of code-level issues that stop AI indexing.
5. Check suppression directives. Search your templates and headers for noindex, nosnippet, max-snippet:0 and data-nosnippet.
6. Check each index separately. Use Google Search Console and Bing Webmaster Tools. In Otterly's test, Google's AI surfaces stopped citing the sites once Google deindexed them, while Copilot, which grounds on Bing, kept citing them.

When you hand fixes to engineering, write each one as a specific, testable ticket. The developer tickets playbook has templates.

• A blanket AI block. One rule meant to stop training often also blocks OAI-SearchBot, PerplexityBot and Claude-SearchBot. That removes search eligibility on those platforms.
• Allowing GPTBot so ChatGPT can find you. GPTBot governs training only. ChatGPT search uses OAI-SearchBot.
• Assuming robots.txt stops user fetchers. OpenAI and Perplexity both say their user-triggered agents may not follow it.
• Trusting robots.txt alone. A CDN rule can return 403 to a bot your robots.txt welcomes. Nothing in your analytics will flag it.
• IP-only allowlisting. OpenAI publishes IP ranges and also warns against relying on IPs alone. Validate on user agent plus your CDN's verified-bot features.
• Reading crawler hits as visibility. A training crawler has no referral surface. Many visits from GPTBot say nothing about appearing in answers.
• Expecting an AI SEO tool or an llms.txt file to fill the gap. Google says Search ignores llms.txt, and no engine documents reading it. Access problems need access fixes.

If you worry that blocking training bots hurts Google rankings, Google states Google-Extended is not a Search ranking signal. More on that in whether Google penalizes blocking AI crawlers.