Copilot Crawler: Bingbot, robots.txt and Your Firewall

Microsoft documents the Bing index as the backing store for Microsoft Copilot, AI summaries in Bing, and Azure's Grounding with Bing Search. Copilot does not keep a separate web of its own in the documentation we hold. If a page is not in Bing's index, Copilot has nothing of yours to ground on.

That makes bingbot the crawler to care about. It fetches pages for the index. Copilot then retrieves from that index at answer time, per query.

Otterly's 2026 experiment shows how tightly the two are linked. Google deindexed both of its AI-generated test sites, but Bing kept them. Of the 3,934 citations Otterly tracked for one site, 2,718 came from Microsoft Copilot, while Google's AI surfaces almost never cited it. An engine cites what its index holds.

The next move: confirm your pages are in Bing at all, using Bing Webmaster Tools. A healthy Google Search Console tells you nothing about Bing. For the wider picture of how pages get into Microsoft's answers, see our guide to Bing and Copilot AI answers.

People search for a "copilot crawler" or a "copilot user agent" expecting a token like OpenAI's OAI-SearchBot. The evidence we hold documents none. Vercel's study of AI crawler traffic excluded Microsoft Copilot because it lacks a unique user agent, so Copilot could not be separated from other Microsoft traffic in the logs.

This has three consequences:

• You cannot allow or block Copilot separately from Bing through robots.txt, based on what Microsoft documents here. Blocking bingbot is the lever, and it takes you out of Bing search too. Read what happens to Copilot if you block Bingbot before you try it.
• Your logs will not show a "Copilot" line. Traffic arrives as bingbot.
• Vercel's finding that ChatGPT and Claude crawlers fetch JavaScript but do not execute it says nothing either way about Microsoft. Copilot was not measured.

Since rendering behavior for Microsoft's pipeline is not established in this evidence, the safe move is the same as for other AI crawlers: put prices, product names and key facts in the HTML before JavaScript runs. Compare this with OpenAI's three separate crawlers, where each setting is independent.

robots.txt rules apply per user agent. A rule for one bot does not carry over to another. The token the evidence names for Microsoft's index crawler is bingbot.

A minimal explicit allow looks like this:

User-agent: bingbot
Allow: /

The common failure is not a deliberate bingbot block. It is a broad rule written for something else. A User-agent: * with Disallow: / on a staging config that reached production removes you from Bing, and with it from Copilot. So does a long Disallow list copied from a template without anyone checking which paths it covers.

Check three things:

1. Fetch your live /robots.txt and find every group that applies to bingbot, including the * group if bingbot has no group of its own.
2. Confirm your pricing, product and about pages are not under a disallowed path.
3. Make sure robots.txt itself loads with a 200. A crawler that cannot read the file cannot follow it.

If you are unsure how names are matched, see whether AI bot names in robots.txt are case-sensitive.

Bing added support for the data-nosnippet HTML attribute in October 2025. Its announcement states that content marked with data-nosnippet "is still indexed normally, but it will be excluded from snippets and AI summaries." The same post says the tagged content "is available for ranking."

So the attribute removes display, not eligibility. The page can still rank. The marked section will not be quoted in a snippet or a Bing AI summary.

<div data-nosnippet>Internal note or legal text here</div>

Bing and Google document four on-page controls for limiting what appears in AI answers: noindex, nosnippet, max-snippet:0 and data-nosnippet. Each is a suppression tool. None of them has an additive opposite. No markup makes a section more likely to be used.

The real risk is accidental suppression. A developer adds data-nosnippet to hide a cookie banner, the attribute lands on a shared wrapper, and the pricing block goes with it. Search your templates and rendered HTML for these four controls, and confirm each one is intentional. Our digest of Bing's data-nosnippet announcement covers the details.

robots.txt is a request. Your firewall, CDN or bot-management layer decides what actually gets served. A site can allow bingbot in robots.txt and still return 403 Forbidden or a JavaScript challenge to it. OpenAI's own guidance notes that bot-mitigation systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors," and the mechanism is the same for any crawler. How often this happens to bingbot is not measured anywhere in the evidence.

As of October 2026, one scheduled change matters. From 2026-09-15, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search stays allowed by default. The evidence does not say which category Cloudflare assigns to bingbot, so check your own dashboard rather than assume.

To check the gate:

1. Request a key page with the bingbot user agent and record the status code.
2. Look for challenge pages or CAPTCHAs returned with a 200, which are a failure too.
3. Review your origin or CDN logs for verified bingbot requests and their responses.
4. Allowlist using your provider's verified-bot programme. Cloudflare's criteria for verified bots include a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS.

The two gates, robots.txt and firewall, must both pass. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Microsoft documents more about its AI pipeline than most vendors, but the gaps are real.

The AI Performance report, in public preview since February 2026 and expanded in June 2026, is the only place in the industry that shows grounding queries: the phrases the AI actually used when it retrieved your content. Microsoft says the grounding-query data is a sample. It frames Citation Share as an observational metric, not a ranking system or competitive scoreboard.

IndexNow's own specification never mentions AI. A 200 response confirms only that the ping arrived.

Bing presence also does not carry over to ChatGPT as a measured effect. OpenAI says ChatGPT search sometimes partners with other providers, and in Otterly's test, ranking in Bing produced effectively zero ChatGPT citations. Treat each engine separately. To see how Copilot currently describes you, try the free Copilot brand check.

Does Copilot have its own user agent I can block in robots.txt? Not in the documentation we hold. Vercel excluded Microsoft Copilot from its crawler study because it lacks a unique user agent. Copilot grounds on the Bing index, which bingbot builds, so the practical robots.txt control is bingbot. Blocking it removes you from Bing search as well as Copilot's index-based answers, so it is a blunt tool rather than a Copilot-only switch.

Does data-nosnippet hurt my Bing rankings? Bing says no. Its October 2025 announcement states that content marked with data-nosnippet is still indexed normally and is available for ranking. It is only excluded from snippets and AI summaries. The risk is applying it by accident to content you want quoted, such as pricing, so audit your templates for the attribute.

Will pinging IndexNow get me into Copilot answers faster? No evidence shows that. Microsoft asserts that IndexNow helps AI systems reference the current version of a page, but no study backs the claim, and the IndexNow specification never mentions AI. A 200 response confirms receipt only. It is reasonable Bing hygiene, with no promised effect on Copilot citations.

If I'm in the Bing index, will Copilot cite me? Not necessarily. Index presence is eligibility, not selection. Copilot retrieves per query and decides what to use. Being in Bing is necessary for index-based Copilot answers, and in Otterly's test Copilot cited sites Bing kept. What you control is access, rendering, snippet controls and stating clear facts on fetchable pages. What Copilot says remains Microsoft's call.

How do I know if my firewall is blocking bingbot? Request a page using the bingbot user agent and check the response. A 403, a 429, or a page containing a challenge or CAPTCHA means the crawler is not getting your content. Then check your CDN or origin logs for verified bingbot requests and their status codes. Your bot-management dashboard shows which categories are blocked, which matters after Cloudflare's September 2026 default changes.