Do I need separate robots.txt rules for each subdomain?

Robots.txt is scoped to one host. RFC 9309, the Robots Exclusion Protocol standard, places the file at the top-level path of a service, and the rules apply to that service only. Google's robots.txt guide says the same: a file covers the host, protocol and port it sits on.

That means docs.example.com, blog.example.com and shop.example.com are three separate hosts. A crawler that wants a page on the docs subdomain requests https://docs.example.com/robots.txt. It never reads the file on example.com for that decision.

What this means for you: if you wrote careful AI bot rules on your main site, they stop at the main site. Your help center, store and blog each run on whatever their own file says, or on nothing. The next move is to list every host you serve and check each file on its own.

Each subdomain file needs the same per-bot thinking as your main file. AI crawlers are not one bot. OpenAI's crawler documentation says GPTBot governs training use and OAI-SearchBot governs whether pages appear in ChatGPT search, and that "each setting is independent of the others". Our digest of OpenAI's crawler documentation covers the full set.

The common mistakes on subdomains are predictable:

• A blanket AI block copied from somewhere. Blocking every AI user agent on your docs subdomain also removes it from ChatGPT search eligibility if OAI-SearchBot is in the list.
• Blocking GPTBot and assuming you are hidden from ChatGPT. You are not. Search eligibility depends on OAI-SearchBot.
• Expecting a token for Google AI Overviews. There is none. As explained in AI Overviews and Googlebot, Googlebot rules are the control there.

User-triggered fetchers behave differently. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules". Treat robots.txt as the index-time control, not a guarantee against every fetch.

A minimal file for a docs subdomain that allows search bots and opts out of OpenAI training might look like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The per-bot robots.txt guide explains each token before you copy anything.

Host scoping is stricter than most people expect. These cases each count as a separate host under RFC 9309 and Google's guide:

• www and the bare domain. www.example.com and example.com are different hosts. If both serve pages, both need a file, or one should redirect to the other.
• http and https. The same hostname on a different protocol is a different service.
• Non-standard ports. example.com:8080 is separate from example.com.
• Vendor-hosted subdomains. A help center or store run by a third-party platform may serve a robots.txt you did not write. Check whether you can edit it at all.
• Staging and preview hosts. These often carry a blanket Disallow: /. That is usually intended, but confirm it never ships to production.

A missing file is not a block. Under RFC 9309, if /robots.txt returns a 4xx status, crawlers may access the whole host. A docs subdomain with no file is open to every bot, including training crawlers you blocked elsewhere.

Path-level control still works inside each host. If you want an AI crawler on some sections of one subdomain but not others, see allowing an AI crawler on some pages but not others.

A correct file on every subdomain does not prove AI crawlers can reach those hosts. OpenAI's help center says eligibility for ChatGPT search requires allowing OAI-SearchBot and confirming that your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Both conditions apply.

Subdomains often sit behind different infrastructure. Your main site may run through one CDN while the shop runs on a commerce platform and the docs on another vendor. Each can have its own bot management settings, and any of them can return a 403 to a crawler your robots.txt allows.

This changed recently for some sites. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts, and existing free-tier customers. A subdomain added after that date can behave differently from your older hosts without anyone editing a file.

What nobody outside the operator controls: whether an allowed, reachable page gets retrieved or cited. Access makes a host eligible. It does not decide the answer.

Run this once, then repeat it whenever a new subdomain launches.

1. List every host. Include www, the bare domain, and every subdomain that serves public pages: docs, help, blog, shop, status, careers. Check DNS records and your sitemap index for hosts you forgot.
2. Fetch /robots.txt on each one. Record the status code. A 4xx means open to all bots. A 200 means read the rules.
3. Resolve the rules per bot. For each host, note what OAI-SearchBot, GPTBot, PerplexityBot, Claude-SearchBot, ClaudeBot, Meta-WebIndexer and Googlebot are allowed to fetch. The step-by-step testing playbook shows how.
4. Compare against intent. Decide which hosts should be in AI search. Docs and help content often should be, as covered in whether AI engines can cite your help center.
5. Test real access. Request a sample page on each host as each AI crawler and record whether you get a 200, a 403 or a challenge page. The AI Crawler Checker does this and shows which crawlers the firewall turns away.
6. Fix and standardise. Start new subdomains from a shared robots.txt template for AI bots so no host launches without rules.