Does Perplexity respect robots.txt?

Perplexity respects robots.txt for one of its two documented agents. The split follows a pattern every major AI platform now uses: one crawler builds an index ahead of time, and a separate fetcher reads pages at the moment a user asks.

• PerplexityBot crawls for Perplexity's search index. Perplexity's crawler documentation says it is "not used to crawl content for AI foundation models." Your robots.txt rule for this agent is the control.
• Perplexity-User fetches a page because a person asked Perplexity a question that needs it. Perplexity's documentation says this fetcher "generally ignores robots.txt rules."

So the honest answer to "does Perplexity respect robots.txt" is: for indexing, yes by its own documentation. For user-triggered fetches, generally no. Our digest of Perplexity's crawler documentation walks through the exact wording.

What it means for you: robots.txt decides whether your pages are eligible for Perplexity's index. It does not decide whether Perplexity can ever read them.

robots.txt rules are per bot. A rule for PerplexityBot affects PerplexityBot only. It does not touch GPTBot, OAI-SearchBot or ClaudeBot, and their rules do not touch it.

To opt out of Perplexity's index, name the agent:

User-agent: PerplexityBot
Disallow: /

To stay eligible while blocking a private area:

User-agent: PerplexityBot
Disallow: /account/

What allowing it does not do

Allowing PerplexityBot makes your pages eligible for its index. It does not make Perplexity retrieve them for a given question, and it does not make Perplexity cite them. Retrieval is decided per query by Perplexity, and nobody outside Perplexity can see that step.

What blocking it does not do

Blocking PerplexityBot does not stop Perplexity-User from fetching a page a user points it at. It also does not remove what other sites say about you, and those pages can still appear in Perplexity answers.

For a full per-agent setup across platforms, use the per-bot guide to robots.txt for AI crawlers or start from the robots.txt template for AI search and training bots.

Perplexity-User acts on behalf of a person in a live session. Perplexity treats that like a browser request rather than a crawl, which is why robots.txt generally does not apply. The difference between the two kinds of agent is covered in index-time crawlers vs answer-time fetchers.

Perplexity is not unusual here. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, its user-initiated agent, for the same reason. Our digest of OpenAI's crawler documentation has the details.

The practical consequence:

If you want Perplexity to be able to answer questions about you, leave both open. If you want no Perplexity access at all, a robots.txt line is not enough on its own.

A firewall can override an allow

A WAF, CDN or bot-management product can return a 403 or a challenge page to a crawler that robots.txt allows. The bot gets nothing, and no dashboard tells you. This is a documented mechanism. How often it happens is not measured. CDN defaults also change: from 15 September 2026 Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. Check how your provider classifies each Perplexity agent rather than assuming.

JavaScript can hide content even when access is open

PerplexityBot was among the AI crawlers found not to execute JavaScript when re-tested as of June 2026, consistent with Vercel's earlier crawler study. If your pricing or product facts only appear after client-side rendering, PerplexityBot may fetch an empty shell. Server-side rendering of the content that matters is the fix.

Spoofed user agents

Anyone can send a request claiming to be PerplexityBot. Cloudflare's bot documentation describes verified bots as those confirmed by signals such as a published IP list with a stable user-agent or reverse DNS. Validate on user-agent plus verification, not the string alone, before you trust or block traffic.

You cannot confirm compliance from outside

Whether a given agent honours your rules shows up only in your own server or CDN logs. A robots.txt file states a preference. It does not enforce one.

1. Decide what you want. Most brands want Perplexity to index and read them. Some want neither. Write that down per agent.
2. Set the robots.txt rule for PerplexityBot to match. Keep it separate from your rules for training bots.
3. Test the network layer. Request a page with each Perplexity user agent and record the status code. The free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
4. If you need Perplexity-User blocked, add a firewall or CDN rule, then confirm it in your logs.
5. Check rendering. Compare raw HTML with the rendered page and make sure key facts are in the HTML before JavaScript runs.

For the side-by-side setup of both agents, including WAF rules, see PerplexityBot vs Perplexity-User: robots.txt and WAFs.

Does blocking PerplexityBot stop Perplexity training models on my content? Perplexity says PerplexityBot is used for search indexing and is "not used to crawl content for AI foundation models." Blocking it is therefore an index decision, not a training decision. If your goal is to stay out of model training, look at the training crawlers other vendors publish, such as GPTBot or ClaudeBot, which are separate agents with separate rules.

Can I block Perplexity-User at all? Yes, at the network layer. Because Perplexity documents that Perplexity-User generally ignores robots.txt, the reliable control is a firewall, CDN or bot-management rule that refuses its requests. Validate on user-agent plus verification rather than the string alone. Expect a trade-off: Perplexity will then fail to read your pages when a user asks about them.

Will allowing PerplexityBot get my site cited in Perplexity answers? No one can promise that. Allowing PerplexityBot makes your pages eligible for the index. Whether Perplexity retrieves a page for a specific question, and whether it cites it, is decided by Perplexity per query. What you control is removing blocks: robots.txt, firewall rules and content hidden behind JavaScript.

Is a firewall block worse than a robots.txt disallow? They do different things. A robots.txt disallow is a stated preference that PerplexityBot follows per Perplexity's docs. A firewall block refuses the request outright, for any agent it matches. The risk with firewalls is accidental blocking: a CDN rule can turn away a crawler your robots.txt allows, and nothing alerts you.

Does Perplexity's API behave the same way as the app? Not necessarily. As of October 2026, web search on Perplexity's Agent API is opt-in, and the older Sonar Chat Completions endpoint retired on 27 September 2026. An API call without web search enabled returns an answer with no live retrieval, so your robots.txt and firewall settings play no part in it.