How do I prepare my website for AI search engines?

An AI engine can only quote a page that passed every gate before the answer was written. The crawler has to be allowed in, reach the server, read the content and find the page in an index. Fix the gates in that order, because a failure early on cancels everything after it.

1. Per-bot robots.txt. Allow the search crawlers you want: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot.
2. Firewall and CDN reachability. Confirm those crawlers get a 200 response, not a 403 or a challenge page.
3. Server-rendered key content. Put prices, product names and definitions in the raw HTML.
4. Snippet controls. Remove any noindex, nosnippet, max-snippet:0 or data-nosnippet you didn't mean to set.
5. Index status. Check Google Search Console and Bing Webmaster Tools separately.
6. Consistent facts. State each key fact once, clearly, and the same way everywhere.

Steps one to five are subtractive. They don't make you more likely to be chosen. They decide whether you can be chosen at all. For more detail on each step, see GEO optimization as a practical order of work.

OpenAI's crawler documentation states that each bot setting is independent. Blocking GPTBot stops training use. Blocking OAI-SearchBot removes you from ChatGPT search answers. The most common mistake is a blanket AI block that removes search eligibility along with training. A per-bot file looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

robots.txt is necessary but not enough. OpenAI's help center says a site must also make sure its host or CDN allows traffic from OpenAI's published search crawler IP ranges. A web application firewall can return 403 to a crawler that robots.txt welcomes. Your analytics won't show it.

This risk grew on 2026-09-15. On that date Cloudflare started blocking Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed by default. A site can lose AI access without anyone editing anything. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Server-render what matters

Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. Googlebot does render. If your pricing loads after JavaScript runs, most AI crawlers see an empty box. Compare your raw HTML with the rendered page and move anything load-bearing into the server response.

Remove accidental suppression

noindex, nosnippet, max-snippet:0 and data-nosnippet are documented controls that keep content out of AI answers. Bing says content marked data-nosnippet is excluded from snippets and AI summaries. These tags often arrive from a template, so check that every one is intentional.

Check both indexes

Google says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. Google also says there are no additional technical requirements. Bing's index backs Copilot. In Otterly's 2,000-post experiment, Google deindexed both test sites and Google's AI surfaces almost stopped citing them, while Bing kept them and Copilot kept citing them. One healthy index tells you nothing about the other.

Most GEO checklists sell tactics that primary sources contradict. Here is where the evidence stands as of October 2026:

• Schema for citations. Google says structured data isn't required for generative AI search. Ahrefs' matched test of 1,885 pages found AI Overview citations fell 4.6% after schema was added. Existing markup does no harm, so there's no need to remove it.
• llms.txt. Google says Search ignores it, and no other engine documents reading it.
• Chunking or AI-friendly rewrites. Google's guide says you don't need to write in a specific way for generative AI search, and that there's no ideal page length.
• E-E-A-T scores and domain authority. Google says E-E-A-T isn't a specific ranking factor. Domain Authority is a Moz metric that no engine computes.
• IndexNow for AI. The specification never mentions AI, and a 200 response only confirms receipt.

The studies behind most rewrite advice tested pages already inside the model's context. The comparison of content rewrites and access fixes and what lab GEO studies show about live engines cover that gap.

You control access, rendering, snippet tags and the facts on your own pages. You don't control what a model learned before its training cutoff, which queries an engine generates, or where your page lands among the retrieved sources. Nobody outside the engine controls those either.

One on-page step has a positive signal: state specific facts plainly. A price, a number or a one-line definition on a fetchable page gives the engine a sentence it can quote. Make sure your pricing page, about page and footer don't contradict each other.

When other sites get cited instead of yours, being named on those sites is associated with appearing in the answer. That route matters most for a brand-new website and is covered in improving AI visibility without changing your site.

Once the gates are fixed, see where you stand. The free checker on this site shows whether AI engines name you for up to three buyer questions. For the full access-first sequence, run a free GEO audit.