Does Google penalize sites that block AI crawlers?

Google does not penalize you for blocking AI crawlers. As of October 2026, it has said so in writing for both of its own AI controls.

First, Google-Extended. Google's common crawlers documentation says this token cuts content use for Gemini Apps and Vertex AI training and grounding. It also says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

Second, the AI features control in Search Console. Google says that opt-out "isn't used as a ranking or inclusion signal affecting other parts of Search."

Third, bots from other companies. robots.txt rules apply per user agent. A Disallow for GPTBot, ClaudeBot or PerplexityBot tells those bots to stay out. Googlebot reads its own group, so Google's ranking systems never see that rule as a decision about Google.

The exception is Googlebot. Block it and you leave Google Search entirely, AI features included. That is not a penalty. It is the crawler doing what you asked.

Google runs AI Overviews and AI Mode on the normal Search index. It says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link, and that "there are no additional technical requirements."

That means there is no separate crawler for AI Overviews. Google says robots.txt directives for Googlebot are the control, because AI is built into Search. So you cannot use robots.txt to leave AI Overviews while staying in regular results. One Googlebot rule governs both.

The practical split looks like this:

For the full contrast, see Google-Extended vs Googlebot.

Google's statements are the best evidence available for Google surfaces. They are also assertions, not studies. Nobody outside Google can test whether a control is truly ignored by ranking systems. Treat them as Google's documented position, which is what you can plan around.

A few scope details matter.

• Google-Extended does not touch AI Overviews. It governs Gemini Apps and Vertex AI. If your goal is to stay out of AI Overviews, this token will not do it. Our page on whether Google-Extended removes you from AI Overviews covers this.
• The AI features opt-out is opt-out, not opt-in. Inclusion is the default for every property. As of October 2026 the toggle is still rolling out to a subset of owners, and child properties inherit the parent's setting by default.
• Opting out of display is not opting out of use. Google says content can still help power Search as a whole, for example in understanding query language.

The AI features control guide walks through where the setting lives and what inherits.

Most "we blocked AI bots and lost Google traffic" stories trace back to a rule that caught Googlebot by accident.

Blanket wildcard blocks

A User-agent: * group with Disallow: / applies to every crawler without its own group, including Googlebot. If you want to block AI training bots only, name them:

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

Snippet controls

nosnippet and max-snippet:0 keep content out of AI answers. They also strip regular text snippets, and Google requires snippet eligibility for AI feature links. Use data-nosnippet on specific sections if you only want to hide part of a page.

Firewalls and CDNs

A bot rule at the network layer can return 403 to a crawler that robots.txt allows. If a security rule meant for AI bots also challenges Googlebot, you drop out silently. Cloudflare changed its defaults on 2026-09-15 to block Training and Agent crawler categories on ad-displaying pages for new domains and free-tier customers, while leaving Search allowed. Check what your edge actually does, as covered in when your CDN or firewall blocks AI crawlers.

Decide what you want, then match each goal to the right control.

1. Stop AI training, keep Google Search: disallow Google-Extended and named training bots. Leave Googlebot alone. The block AI training but stay in AI search playbook lists the tokens per vendor.
2. Stay out of AI Overviews and AI Mode, keep blue links: use the Search Console AI features control if your property has it, or snippet controls on specific content.
3. Audit your robots.txt group by group. Confirm Googlebot is not caught by a wildcard. The robots.txt Checker for AI Crawlers shows how each crawler resolves.
4. Test the network layer. robots.txt permission means nothing if your firewall returns 403.

None of these steps guarantees rankings or citations. They make sure your own rules are not removing you.