Site Migration Checklist for AI Crawler Access

A migration swaps out the layer AI crawlers touch first. You get a new host, a new CDN zone, a new framework and new templates. Each one can carry a block the old site never had, and none of them announces itself.

The breaks cluster in five places:

• robots.txt copied from staging, where everything was disallowed.
• CDN and firewall rules on a fresh zone, with bot defaults you never chose.
• Rendering mode, when the new framework builds content in the browser.
• noindex leftovers from staging templates or headers.
• Snippet directives such as nosnippet or data-nosnippet inherited by a shared layout.

All five sit at the access stage, which is the one stage you control. Retrieval (whether an engine pulls your page for a query) and generation (whether it cites you) belong to the engine. A clean access check makes your pages eligible. It does not make anyone cite them. For the wider picture, see whether a migration can break your AI visibility.

Google is a special case. Its AI features documentation says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode, with no additional technical requirements. There is no separate robots.txt token for those surfaces, so Googlebot rules and index eligibility do the work.

You can't tell what the migration broke unless you know what worked before it. Spend an hour on the old site first.

1. Pick your fact pages. List the URLs that state what you sell and what it costs: homepage, pricing, main product pages, about, and your most visited docs.
2. Save the live robots.txt. Note the rule that applies to each of OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, ClaudeBot, Meta-WebIndexer, Googlebot and bingbot.
3. Request each fact page as each crawler. Record the status code and whether a challenge page came back instead of content. This guide on checking whether AI crawlers can reach your site covers the method.
4. Fetch the raw HTML with JavaScript off. Confirm your prices, product names and headings are in the response before any script runs.
5. Record indexing and snippet directives. Note every meta name="robots" tag, X-Robots-Tag header and data-nosnippet attribute on those pages.
6. Screenshot index status in Google Search Console and Bing Webmaster Tools for the same URLs.

Then run steps 2 to 5 against staging. Staging should be locked. The point is to write down exactly where the lock lives so someone removes it at launch.

Work through this list before sign-off and again after go-live.

robots.txt

• The production file is not the staging file. A blanket rule like this one must not ship:

User-agent: *
Disallow: /

• Search bots are allowed if you want to appear in AI search. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
• Training bots are a separate decision. OpenAI says each crawler setting is independent, so blocking GPTBot does not remove you from ChatGPT search, and allowing it does not put you there.
• A labelled example of search allowed, training blocked:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

CDN and firewall

• The new zone does not challenge or 403 AI crawlers. OpenAI's help page says eligibility needs OAI-SearchBot allowed and a host or CDN that accepts traffic from its published IP addresses.
• On Cloudflare, check AI crawler categories. From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and new sites on existing accounts, which is exactly what a migration creates.
• robots.txt itself returns 200 to every crawler.

Rendering

• Fact pages carry their key content in the initial HTML. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. Googlebot renders; most AI crawlers do not. If you are picking a build mode, compare SSR, prerendering and static generation.
• Content inside tabs, accordions or load-more buttons is present in the raw HTML, not injected on click.

Indexing directives

• No noindex in templates, meta tags or X-Robots-Tag headers on public pages.
• Canonicals point to the new URLs, and redirects from old URLs are single hops.

Snippet directives

• No stray nosnippet or max-snippet:0 in the shared layout.
• data-nosnippet wraps only what you mean to hide. Bing states that content marked this way is still indexed but excluded from snippets and AI summaries.

Search Console and Bing

• A new domain means a new property. Verify it in both tools and check the generative AI inclusion control, which child properties inherit from their parent.

Run the full baseline again on production, the same day DNS or the CDN switches. Cache layers and edge rules often differ from staging, so a clean staging test proves little.

1. Fetch production robots.txt and diff it against your saved baseline.
2. Request every fact page as each AI crawler and compare status codes with the baseline. The free AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.
3. Diff raw HTML against the rendered page for each fact page. Any price or product name that appears only after rendering is invisible to most AI crawlers.
4. Grep for noindex, nosnippet, max-snippet and data-nosnippet across templates and response headers.
5. Request indexing of key URLs in Search Console and Bing Webmaster Tools, and confirm old URLs redirect.
6. Log the results with a date, so the next check has something to compare against.

If you'd rather script steps 1 to 4, these AI SEO scripts you can run yourself cover the fetch, user-agent and diff work.

A bad result on one bot matters even if the others pass. A 403 to OAI-SearchBot alone is enough to drop you from ChatGPT search. If robots.txt looks fine and you still never appear, work through what else can block ChatGPT when robots.txt allows AI bots.

Treat launch day as the first run, not the only one.

• Launch day: the full audit on production.
• About a week later: once caching, edge rules and redirects have settled.
• After any change to the CDN, firewall, framework, templates or robots.txt.
• On a regular schedule even when nothing changed on your side. Provider defaults can shift under you, as Cloudflare's September 2026 change showed.

Fold these checks into your routine AI SEO checklist so access gets tested as often as content does.

A migration is a tempting moment to add every AI tactic you've read about. Keep the launch list to things that can block you.

Adding schema or an llms.txt file is not an access fix. Google says structured data isn't required for its generative AI features, and that Google Search ignores llms.txt. Removing existing schema isn't supported either; Google says it does no harm.

One content check is worth adding. Migrations often leave old and new prices or product names live on different pages. State each fact once, clearly, in HTML a crawler can fetch, and run a check for contradicting facts on your own site after launch. Once access is confirmed, the free checker on this site shows how AI answers describe you for up to three buyer questions.

Should I block GPTBot during a migration? Only if you want to keep content out of OpenAI's model training. GPTBot governs training. OAI-SearchBot governs ChatGPT search, and OpenAI says each setting is independent of the others. Blocking GPTBot while you migrate will not hide staging from ChatGPT search, and it will not help or hurt your search eligibility. Lock staging with authentication or a blanket disallow, then remove that lock at launch.

How long after a migration do AI answers reflect the new site? Nobody publishes a number. Answers with live retrieval depend on when each engine recrawls and reindexes your pages. Answers without retrieval come from the model's frozen knowledge and will not change until a new model ships; OpenAI lists a February 16, 2026 knowledge cutoff for its GPT-5.6 models. Your job is to make sure every crawler can fetch the new pages from day one.

What if my CDN blocks a crawler that robots.txt allows? The crawler is still blocked. robots.txt states permission, but the CDN or firewall decides whether the request gets through. OpenAI's help page says eligibility for ChatGPT search needs both OAI-SearchBot allowed in robots.txt and a host or CDN that accepts traffic from its published IP ranges. Fix it in your bot management settings, then re-request the page as that crawler to confirm.

Does moving to a JavaScript framework hurt AI visibility? It can, if the framework renders content only in the browser. Vercel found that ChatGPT and Claude crawlers download JavaScript files but do not execute them, so client-rendered prices and product details never reach them. Googlebot does render JavaScript. Use server-side rendering, prerendering or static generation for pages that carry your key facts, and check the raw HTML before launch.

Do I need a new Search Console property after a domain change? Yes. A new domain is a new property in Google Search Console and a new site in Bing Webmaster Tools. Verify both, submit sitemaps, and check index status for your key URLs. In Search Console, also check the generative AI inclusion control. Inclusion is the default, but child properties inherit from their parent, so a setting made elsewhere can carry over.