Perplexity's Crawler Documentation, Digested

Perplexity maintains a crawler reference page in its developer documentation. The version our evidence base verified is dated 29 January 2026. It names two agents and gives each a distinct job.

• PerplexityBot crawls the web to build and refresh Perplexity's search index.
• Perplexity-User fetches a page at answer time, when a user's request needs it.

This split matches what OpenAI and Anthropic publish: an index-time crawler that reads robots.txt, and an answer-time fetcher that behaves differently. If you manage access for several AI platforms, read this alongside OpenAI's crawler documentation and Anthropic's crawler documentation. The distinction itself is covered in index-time crawlers vs answer-time fetchers.

This digest is AI-assisted. It was written from our verified evidence base, which records what each primary source actually says. We did not run crawl tests, interview Perplexity, or measure traffic for this page.

We read the Perplexity page for three things:

1. What each agent is for, in Perplexity's own words.
2. Whether each agent honours robots.txt.
3. What the page does not say, because the gaps decide what you have to check yourself.

We then set it against other primary sources on rendering and network access, since robots.txt is only one of the gates a crawler passes through.

Two quoted lines carry the whole page.

PerplexityBot is for search, not training

Perplexity states that PerplexityBot is used for search indexing and is "not used to crawl content for AI foundation models." That makes it a retrieval bot, not a training bot. Blocking it has a search consequence: your pages become ineligible for Perplexity's index. It is not a way to opt out of model training, because Perplexity does not describe it as a training crawler.

Perplexity-User generally ignores robots.txt

Perplexity says of Perplexity-User that "this fetcher generally ignores robots.txt rules." The reasoning is the same one OpenAI gives for ChatGPT-User: a person triggered the fetch, so the operator treats it as a user action. OpenAI says robots.txt "may not apply" to ChatGPT-User. Perplexity's wording is firmer.

What follows

robots.txt controls index-time eligibility for Perplexity. It does not reliably control answer-time fetching. If your goal is to keep content out of Perplexity entirely, a robots.txt disallow is not enough. If your goal is to be included, an allow is necessary but nowhere near sufficient. For a short version of this answer, see does Perplexity respect robots.txt.

The page is a statement of intent from the operator. It is not a measurement, and it leaves several things open.

• No effect on citation. Allowing PerplexityBot makes you eligible. Perplexity does not say it improves your chance of being retrieved or cited. Only Meta's crawler page, in the whole first-party corpus, ties crawler access to citation.
• No compliance data. Cloudflare, which sees the network traffic, says some site owners "might be unsure whether these bots even respect robots.txt rules." Nobody has published how often robots.txt is honoured in practice.
• Nothing about your firewall. A CDN or bot-management rule can block a crawler that robots.txt allows. The documentation cannot tell you whether that is happening on your site.
• Nothing about rendering. Vercel reported that major AI crawlers fetch JavaScript files but do not execute them, and our evidence base records PerplexityBot among the crawlers independently re-confirmed as not rendering JavaScript as of June 2026. Content that appears only after JavaScript runs may be invisible to it.
• Nothing about the API. On Perplexity's Agent API, web search is opt-in. An answer from that API with search off never touches your site at all.

A Perplexity crawler has to pass three gates before your page can enter its index:

1. robots.txt permission for PerplexityBot.
2. Network reachability: your host, CDN and firewall must return the page, not a 403, a 429 or a challenge.
3. Readable content: the text that matters must be in the HTML before JavaScript runs.

Most sites check only the first. The second is where silent failures happen. OpenAI's guidance names the same pattern for its own bots, warning that bot-management systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." From 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed by default. Defaults like these can change a site's AI access without anyone on your team editing a file.

The point is not that your firewall is blocking Perplexity. It is that you cannot know from robots.txt alone. The PerplexityBot vs Perplexity-User guide covers the WAF side in more depth.

Start with the intent you actually have, then check each gate.

If you want to appear in Perplexity

Give PerplexityBot an explicit rule rather than relying on a wildcard:

User-agent: PerplexityBot
Allow: /

Then confirm three things:

• Robots.txt resolves per agent. A blanket AI block written for training bots often catches retrieval bots too. Use a robots.txt template for AI bots that separates the two.
• The crawler gets a 200. Request a real content URL with the PerplexityBot user-agent and record the status code. A 403 or a challenge page means the firewall is overriding your robots.txt. In your CDN, check how verified bots are classified. Cloudflare's criteria include a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS.
• Key facts are in the raw HTML. Fetch the page with JavaScript off. If your pricing or product names disappear, server-render them.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

If you want to stay out

Disallow PerplexityBot in robots.txt to leave the index. Accept that Perplexity-User may still fetch pages a user asks for. If that matters, the control lives at the network layer, not in robots.txt, and it is worth reading whether all AI crawlers obey robots.txt before you decide.