Should I block AI crawlers I don't recognize?

The right decision depends on what the bot does, not on whether you recognize its name. Most large AI operators split crawling into three jobs, and each job has a different consequence when you block it.

• Training crawlers collect pages that may be used to train models. Blocking one opts your content out of training. It does not remove you from that company's live search answers.
• Search crawlers build the index an AI engine retrieves from at answer time. Blocking one removes your pages from that engine's answers.
• User-triggered fetchers load a page because a person asked the assistant to. Several operators say robots.txt may not apply to these at all.

OpenAI is the clearest example. Its bots page says GPTBot crawls content that may be used for training, while OAI-SearchBot surfaces sites in ChatGPT search, and "each setting is independent of the others." Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." So a blanket block on every AI-sounding name can quietly cost you search eligibility you wanted to keep.

The decision rule is simple. Find the operator, read its documentation, and match the bot to one of the three jobs. Then block or allow on purpose. If you want the wider strategic question first, see whether to let AI crawlers access your site at all.

Work through these steps when a new name shows up in your server logs.

1. Read the full user agent string. Most legitimate bots include a product token and often a URL pointing to the operator's crawler page.
2. Find the operator's documentation. Search the operator's own developer or help site, not a third-party list. The digests of OpenAI's crawler documentation, Perplexity's crawler documentation and Anthropic's ClaudeBot, Claude-SearchBot and Claude-User cover the most common names.
3. Assign the job. Training, search or user-triggered. If the documentation does not say, record it as unknown.
4. Check whether it publishes IP ranges or supports reverse DNS. OpenAI, for example, publishes OAI-SearchBot ranges at openai.com/searchbot.json.

Here is how the documented bots sort, as of October 2026:

Perplexity states PerplexityBot is not used to crawl content for AI foundation models. Meta says allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses." That is one of the few first-party statements tying crawler access to citation.

robots.txt is a request. It works only when the crawler chooses to honor it. Cloudflare, which sees bot traffic at network scale, says site owners "might be unsure whether these bots even respect robots.txt rules, especially newer or less transparent crawlers." That is a plain admission that compliance is not measured.

The documented operators add their own exceptions. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user starts the action. Perplexity says Perplexity-User "generally ignores robots.txt rules." Meta documents that Meta-ExternalFetcher may bypass them too. These are fetchers acting for a person, which the guide on user-triggered AI fetchers covers in more depth.

What it means: if you truly need a bot kept out, write the robots.txt rule and also enforce it at the firewall or CDN. Block by verified identity where possible, not just by name.

The reverse risk is real too. A firewall can block a bot your robots.txt allows. OpenAI says eligibility for ChatGPT search requires allowing OAI-SearchBot and confirming your host or CDN allows its published IP addresses. As of September 15, 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers. You can lose access without editing anything.

The name is real but the bot is not

Anyone can send a request claiming to be GPTBot or PerplexityBot. Blocking or allowing purely by user agent string trusts the claim. Cloudflare's verified-bot criteria rely on a cryptographic signature, a published IP list with a stable user agent, or reverse DNS. The playbook on verifying real AI crawlers versus spoofed user agents walks through the checks.

No documentation exists

If you cannot find an operator page, you cannot know the bot's job or whether it honors your rules. Treat it as unverified. Blocking it at the firewall is a reasonable default, because there is no documented benefit you would be giving up.

The bot is not named in your robots.txt

An unlisted bot falls under your User-agent: * group. Read what happens when robots.txt doesn't mention an AI bot before assuming it is blocked or allowed.

Known names with thin documentation

Some crawlers, such as Bytespider, appear often in logs but sit outside the main operators' documentation. Decide on the same rule: no clear job, no clear benefit, enforce at the edge if you want it out.

Keep the decision deliberate and per bot. A short robots.txt that separates training from search looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Then work through this list:

1. Pull a week of server or CDN logs and list every bot user agent.
2. Classify each one from the operator's documentation. Mark undocumented bots as unverified.
3. Write per-bot robots.txt rules for the documented ones, keeping the search crawlers you want.
4. Enforce blocks on unverified bots at the firewall, using IP ranges or reverse DNS where available.
5. Confirm the search crawlers you allowed actually get a 200 response, not a 403 or a challenge page.

Step 5 is where most sites never look. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. Allowing a crawler buys eligibility only. It does not guarantee you will be retrieved or cited.