PerplexityBot vs Perplexity-User: robots.txt, firewalls and the 403 problem

99Visibility Research · Updated October 21, 2026 · 7 min read

Short answer

Perplexity runs two crawler user agents. PerplexityBot crawls pages for Perplexity's search index, and Perplexity's documentation says it is not used to crawl content for AI foundation models. Perplexity-User fetches pages when a user's request triggers it, and Perplexity says it generally ignores robots.txt rules. robots.txt is only one gate. A firewall or CDN can still return a 403 to PerplexityBot when robots.txt allows it.

Key takeaways

PerplexityBot is Perplexity's search indexing crawler, and Perplexity documents that it is not used to crawl content for AI foundation models.
Perplexity-User is a user-triggered fetcher that, by Perplexity's own documentation, generally ignores robots.txt rules, so a Disallow line is not a reliable block.
A robots.txt Allow for PerplexityBot buys eligibility only if your firewall, CDN and bot management also let the request through.
Test what PerplexityBot actually receives (status code, challenge page, robots.txt reachability) instead of trusting your robots.txt file alone.
Allowing PerplexityBot makes you eligible for Perplexity's index. It does not guarantee retrieval, citation or placement in any answer.

The two Perplexity user agents, side by side

Perplexity documents two user agents, and they do different jobs. Treating them as one bot is the most common source of confusion.

User agent	What Perplexity says it does	Does robots.txt control it?
`PerplexityBot`	Crawls pages for Perplexity's search index. Perplexity states it is "not used to crawl content for AI foundation models"	Yes, this is the token your robots.txt rules govern
`Perplexity-User`	Fetches a page when a user's request triggers it	Perplexity says it "generally ignores robots.txt rules"

The split matches a pattern across AI platforms: an index-time crawler that builds a store of pages ahead of time, and an answer-time fetcher that runs on demand. Our comparison of index-time crawlers and answer-time fetchers covers why the two behave so differently. For the full wording of Perplexity's page, see Perplexity's crawler documentation, digested.

What Perplexity's documentation, as summarised in our evidence base, does not give you: any statement about how often PerplexityBot recrawls, how its index ranks pages, or how a page gets picked for a given answer. Treat anything beyond the two definitions above as unknown.

PerplexityBot and robots.txt

PerplexityBot reads robots.txt rules addressed to its own token. A minimal rule that allows it looks like this:

User-agent: PerplexityBot
Allow: /

Three things follow from how robots.txt works per bot.

Rules don't transfer between vendors. Allowing OAI-SearchBot or Claude-SearchBot says nothing to PerplexityBot. Each search crawler needs its own decision.
Training and search tokens are separate. PerplexityBot is the search crawler. Blocking a training crawler such as GPTBot or ClaudeBot has no bearing on it. The reverse mistake is more costly: a blanket "block all AI bots" rule that sweeps in PerplexityBot removes your eligibility for Perplexity's index, often without anyone meaning to.
Allow means eligible, not chosen. Letting PerplexityBot in lets it fetch and store your pages. It does not decide whether Perplexity retrieves them for a question.

The next move: open your live robots.txt, find every group that could match PerplexityBot (its own group, or User-agent: *), and confirm the result is the one you intended.

Why Perplexity-User generally ignores robots.txt

Perplexity-User runs because a person asked Perplexity something that needs a page fetched. Perplexity's documentation says this fetcher "generally ignores robots.txt rules." That is the vendor's own statement, not an outside accusation.

Perplexity is not alone here. OpenAI says robots.txt rules "may not apply" to ChatGPT-User for the same reason: the action is user-initiated. The OpenAI crawlers page walks through that equivalent.

What this means in practice:

A Disallow line for Perplexity-User is not a reliable control. Don't rely on it to keep a page out of Perplexity answers.
If you need to stop answer-time fetches, the decision moves to the network layer (your firewall, CDN or bot management), not robots.txt.
If you want Perplexity answers to be able to read your pages on demand, a firewall rule that challenges or blocks Perplexity-User undoes that, whatever robots.txt says.

The word "generally" matters. Perplexity does not publish the exceptions, so you can't plan around them.

How a firewall blocks PerplexityBot when robots.txt allows it

robots.txt is a request. Your firewall is enforcement. They are configured in different places, often by different people, and nothing forces them to agree. That is why robots.txt and the firewall are two separate gates.

The failure looks like this. Your robots.txt allows PerplexityBot. A web application firewall, CDN rule or bot-management product classifies the request as automated traffic and answers with a 403 Forbidden, a rate-limit 429, or a JavaScript challenge or CAPTCHA page. An automated crawler can't solve a challenge, so it gets nothing usable. No dashboard tells you this happened.

The mechanism is documented by the operators: OpenAI's advertiser guidance warns that these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." How often it happens across the web has not been measured by anyone in our evidence base.

A dated change to watch

From 2026-09-15, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads, while Search stays allowed by default. This applies to new domains, new sites on existing accounts and all existing free-tier customers. A site can change its AI crawler access without anyone editing a file. As of October 2026, check which category your CDN assigns to PerplexityBot and to Perplexity-User, because the two may be treated differently.

Don't forget robots.txt itself

If the firewall blocks the crawler from /robots.txt, the bot can't even read your rules. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file," and the same logic applies to any crawler. Our guide to making sure crawlers can fetch robots.txt itself covers the check.

How to test what Perplexity actually receives

Treat your robots.txt as a statement of intent, then verify the response a crawler gets.

Request a real content page with the PerplexityBot user agent. Use a URL that matters, such as pricing or a product page, not just the homepage.
Record the status code. 200 passes. 403 points to a bot-mitigation block. 429 is a rate limit.
Read the body, not just the code. A 200 that returns a challenge or CAPTCHA page is a failure.
Test from more than one network. Rules often differ for datacenter and residential traffic, so one passing request can hide a block.
Request /robots.txt the same way. If it fails, the bot can't read your rules.
Repeat for Perplexity-User to see whether answer-time fetches get through.
Check your CDN's bot settings for the Search, Agent and Training categories, and look in your origin or CDN logs for what Perplexity's agents actually received.

If you'd rather not script this, the AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

When a request is blocked, the fix sits in the firewall or CDN: an allow rule for the crawler, based on the vendor's verification method rather than the user-agent string alone, since user agents are easy to spoof.

What allowing PerplexityBot gets you, and what it doesn't

Passing both gates makes your pages eligible for Perplexity's index. That is the part you control. Everything after it belongs to Perplexity.

Retrieval is per question. Perplexity decides for each query which pages enter the answer. Nobody outside Perplexity can see that decision.
Rendering still matters. Vercel's study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, and independent re-testing in June 2026 found the same for PerplexityBot. Content that only appears after JavaScript runs may be invisible to it. Put prices, product names and key facts in the server-rendered HTML.
No markup shortcut exists. Schema, llms.txt and "AI-friendly" rewrites have no credible evidence behind them as citation levers. A critical survey of 45 GEO studies found no technique with a stable, cross-platform causal effect on discoverability.

The honest scope: you can remove the blocks. You can't make Perplexity cite you.

Frequently asked questions

Does blocking PerplexityBot stop Perplexity from training models on my content?

Perplexity's documentation says PerplexityBot is not used to crawl content for AI foundation models, so blocking it is a search decision, not a training opt-out. Blocking it removes your eligibility for Perplexity's search index. If your goal is limiting AI training, the relevant tokens belong to other vendors' training crawlers, such as GPTBot or ClaudeBot, each set separately.

Can I block Perplexity-User with robots.txt?

Not reliably. Perplexity states that Perplexity-User generally ignores robots.txt rules because a user's request triggers the fetch. A Disallow line may be honoured sometimes, but Perplexity does not publish when. If you must stop answer-time fetches, the control sits in your firewall, CDN or bot-management settings. Weigh that against losing the ability for Perplexity answers to read your pages on demand.

Does a 403 for PerplexityBot always mean my firewall blocked it?

A 403 means the server refused the request, and bot-mitigation systems are a common cause. It can also come from hosting rules, geographic restrictions or application settings. Check your CDN and WAF logs for the request first. If robots.txt allows PerplexityBot and the response is still 403, the block sits somewhere in your network stack, not in robots.txt.

Do I need a separate robots.txt group for PerplexityBot?

Only if you want it treated differently from your `User-agent: *` rules. If the wildcard group already allows what you want PerplexityBot to fetch, it applies. Many sites add an explicit PerplexityBot group so the decision is visible and doesn't change by accident when someone edits the wildcard rules or adds a blanket AI block.

Does Perplexity read llms.txt?

No engine in our evidence base, Perplexity included, documents parsing llms.txt. Google says Google Search ignores it. Publishing one does no harm, but there is no evidence it affects whether Perplexity indexes or cites your pages. Your time is better spent on the checks that have documented effects: robots.txt permission, firewall reachability and server-rendered content.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.