Can I allow an AI crawler on some pages but not others?

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

Yes. robots.txt lets you write Allow and Disallow rules by path for each named crawler, such as OAI-SearchBot, GPTBot or PerplexityBot. To keep part of a page out of AI answers while the page stays crawlable, use data-nosnippet or snippet directives. Two limits apply: user-triggered fetchers like ChatGPT-User and Perplexity-User may not honor robots.txt, and a firewall can block a bot your rules allow.

Key takeaways

robots.txt rules are grouped per user agent, and each group can allow some paths and disallow others.
When several rules match a URL, the most specific (longest) path wins, and a crawler with its own group ignores the * group.
data-nosnippet, nosnippet and max-snippet keep content out of AI display without blocking the crawl.
User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt, so it is not a privacy control.
Allowing a path makes it eligible for an AI engine. It does not make that engine cite it.

Yes: write path rules inside each bot's group

robots.txt is organized into groups. Each group starts with one or more User-agent lines and lists the paths that agent may or may not fetch. The Robots Exclusion Protocol (RFC 9309) defines both Allow and Disallow, so you can open most of a site to one crawler and close specific directories or files.

AI operators publish separate user agents for separate jobs. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for user actions, and states that each setting is independent of the others. That lets you give each bot its own map of the site.

Say you run a software company and want ChatGPT search to reach your product and docs pages, keep it out of a customer portal, and keep your research reports out of training:

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/

User-agent: GPTBot
Disallow: /research/
Disallow: /account/

User-agent: *
Disallow: /account/

The same pattern works for PerplexityBot, Claude-SearchBot, ClaudeBot and Meta-WebIndexer. The per-bot breakdown of which agent does what lives in our guide to robots.txt for AI crawlers.

How a crawler decides which rule applies

Two rules from RFC 9309 decide most outcomes.

Group selection. A crawler follows the group that names it. If OAI-SearchBot has its own group, it ignores everything under User-agent: *. Repeat any shared rules (like the /account/ block above) inside each named group.
Longest match wins. When an Allow and a Disallow both match a URL, the rule with the longer path applies. Allow: /research/summary/ beats Disallow: /research/ for pages under the summary folder. If two matching rules are equally long, Allow wins.

Paths are case-sensitive and match from the start of the URL path. Each host also needs its own file: rules at www do not cover a docs or app subdomain. See separate robots.txt rules for each subdomain for that case, and how Allow and Disallow interact for more worked conflicts.

For Google, there is no separate robots.txt token for AI Overviews or AI Mode. Googlebot rules govern all Google Search surfaces, so a Disallow for Googlebot on a path removes it from classic results too. Our page on AI Overviews and Googlebot covers the Search Console control that exists instead.

Keeping a section out of AI answers without blocking the page

Sometimes the page should be crawled, but one block should not appear in an answer: internal pricing notes, a legal disclaimer, a stale promotion. robots.txt cannot do this because it works per URL. On-page controls do.

data-nosnippet on an HTML element. Bing said in October 2025 that content so marked "is still indexed normally, but it will be excluded from snippets and AI summaries." Google documents the same attribute in its robots meta tag specification.
nosnippet or max-snippet:0 in a robots meta tag, to stop snippet display for the whole page.
noindex, to keep the page out of the index entirely.

Google's AI features guide says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. So these controls remove eligibility on Google's surfaces. They are suppression tools only. No markup makes a section more likely to be used.

Check templates before you ship. A data-nosnippet added to hide a cookie banner can end up wrapped around your main content.

Where path rules stop working

robots.txt is a request to cooperating crawlers, not an access control. Three gaps matter.

User-triggered fetchers. As of October 2026, OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a person initiates the request. Perplexity says Perplexity-User "generally ignores robots.txt rules." If a path must stay private, put it behind a login, not a Disallow line.

The network layer. A firewall or CDN can return 403 to a bot your robots.txt allows. OpenAI's help center says eligibility for ChatGPT search needs both: allow OAI-SearchBot and confirm your host or CDN accepts traffic from its published IP addresses. Cloudflare also changed its AI crawler defaults on 15 September 2026 for new domains and free-tier sites, so access can change without anyone editing your rules. If your rules look right but you never appear, what else can block ChatGPT walks through the checks.

Eligibility is not selection. Allowing a path makes it fetchable. Whether an engine retrieves and cites it for a given question is decided by the engine, and nobody outside controls that.

What to do next

List which sections each bot should reach. Decide search bots and training bots separately.
Write a named group per bot, repeating shared rules in each one.
Test each rule against real URLs for each user agent. The per-bot testing playbook shows how.
Confirm the bots actually get a 200 response. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
Use data-nosnippet for sections that should never be quoted, and audit templates for accidental use.
Gate anything truly private behind authentication.

If your main goal is opting out of training while staying in AI search, follow block AI training but stay in AI search.

Frequently asked questions

Can I block AI training on one directory but keep it in AI search?

Yes. Disallow the directory under GPTBot, ClaudeBot or other training agents, and leave it allowed under OAI-SearchBot, Claude-SearchBot and PerplexityBot. OpenAI states its settings are independent, so blocking GPTBot does not remove a page from ChatGPT search. For Gemini apps, Google-Extended is the separate training and grounding control, and it does not affect Google Search inclusion.

Does a Disallow for Googlebot only remove a page from AI Overviews?

No. Google has no AI-specific robots.txt token, so a Googlebot Disallow applies to all Google Search surfaces, including classic results. If you want a page in search results but not quoted, use nosnippet, max-snippet or data-nosnippet instead. Google's guide says AI Overviews and AI Mode links need a page that is indexed and snippet-eligible.

Will data-nosnippet hurt my rankings?

According to Bing's October 2025 announcement, content marked data-nosnippet is still indexed normally and remains available for ranking. It is only excluded from snippets and AI summaries. The risk is accidental use: if a template wraps your main content in the attribute, that content cannot be shown in AI answers even though the page ranks.

If I allow a path for every AI bot, will it get cited?

Not necessarily. Allowing a bot gives the page eligibility to be fetched and indexed. Retrieval and citation are decided per question by each engine, and no published study shows a reliable way to force them. What you control is removing blocks: robots.txt rules, firewall responses, JavaScript-only content and snippet directives.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.