Does Perplexity have its own search index?

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

Yes. Perplexity documents PerplexityBot as the crawler that collects pages for its search index, and Vespa describes Perplexity building its AI search on Vespa.ai. A separate fetcher, Perplexity-User, retrieves pages at answer time and, per Perplexity, generally ignores robots.txt. To be eligible for Perplexity's index, your robots.txt must allow PerplexityBot and your firewall or CDN must let it through.

Key takeaways

Perplexity runs its own search index, filled by PerplexityBot, and Vespa describes the infrastructure it is built on.
PerplexityBot is for search indexing, and Perplexity says it is not used to crawl content for AI foundation models.
Perplexity-User fetches pages when a user's request needs them and generally ignores robots.txt, so blocking it there is not reliable.
A robots.txt Allow does nothing if your firewall or CDN returns a 403 or a challenge page to the crawler.
Passing both gates makes you eligible for the index. It does not guarantee retrieval or citation for any query.

The full answer: Perplexity crawls and indexes the web itself

Perplexity operates its own search index. Its crawler documentation names PerplexityBot as the agent that crawls pages for that index, and states the bot is not used to crawl content for AI foundation models. Vespa, the search engine company, has published a write-up describing Perplexity building its AI search at scale on Vespa.ai.

That matters because it changes where you need to be visible. Ranking in Google or Bing does not put you in Perplexity's index. PerplexityBot has to fetch your pages itself, and that only happens if two gates are open:

robots.txt must allow PerplexityBot.
Your network layer (firewall, CDN, bot management) must actually serve the page to it.

Perplexity also runs a second agent, Perplexity-User, which works differently. Understanding the split between the two is most of what you need to know. For the model side of the product, see which models Perplexity uses.

Two agents, two different controls

Perplexity's two agents sit on opposite sides of a distinction worth learning: index-time crawlers versus answer-time fetchers.

Agent	When it runs	What robots.txt does
PerplexityBot	Ahead of time, to build the search index	Controls it. Disallow keeps your pages out of the index
Perplexity-User	At answer time, when a user's request needs a page	Perplexity says this fetcher generally ignores robots.txt rules

PerplexityBot is the one you manage for discoverability. A minimal rule that allows it looks like this:

User-agent: PerplexityBot
Allow: /

Perplexity-User is user-triggered, so Perplexity treats it more like a browser acting for a person. If you want to keep content away from it, robots.txt is not the tool. Network-level rules are, and they carry their own trade-offs. The full breakdown is in PerplexityBot vs Perplexity-User: robots.txt and WAFs.

The firewall gate robots.txt cannot see

robots.txt is a request, not a lock. Your CDN or firewall decides what a crawler actually receives, and it can block a bot your robots.txt allows. Bot-management systems can return a 403, a rate-limit response, or a JavaScript challenge that an automated crawler cannot pass. Cloudflare's bot documentation describes how it classifies and verifies bots, which is the layer where these decisions happen.

This failure is silent. Your robots.txt reads correctly, nothing in your analytics flags it, and PerplexityBot still gets nothing indexable. Defaults can also change without you editing anything: from 2026-09-15, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stays allowed by default.

There is a third, quieter gate. Most AI crawlers tested by Vercel and later re-checked in June 2026, PerplexityBot included, do not execute JavaScript. Content that appears only after client-side rendering may never reach the index even when the request succeeds.

What allowing PerplexityBot does and does not do

Opening both gates buys eligibility. It does not buy retrieval, ranking or citation.

Retrieval is per query. Perplexity decides which indexed pages enter the context for a given question. No publisher controls that step.
API answers can skip the web entirely. As of October 2026, web search is opt-in on Perplexity's Agent API, and the older Sonar Chat Completions endpoint retired on 2026-09-27. Unmigrated code can return answers with no web grounding, so a test run through the API may not reflect the app.
Citations point to whole sources. Perplexity cites sources by an integer index rather than quoting text spans. More on that in Perplexity citations explained.
On-page tricks lack evidence. A 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found no technique with a stable, cross-platform causal effect on discoverability.

One on-page move does have a positive signal behind it: stating specific facts, such as a price or a one-line definition, in HTML that loads without JavaScript. That gives the engine a sentence it can use if it retrieves the page.

What to do next

Treat this as an access audit, not an optimization project.

Read your robots.txt per agent. Confirm PerplexityBot is allowed. Check that a blanket AI block meant for training bots is not catching it.
Request a page as PerplexityBot. Look at the status code and the body. A 200 with real content passes. A 403, 429 or challenge page fails. The free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
Check your CDN's AI crawler settings. If you use Cloudflare, review the bot category defaults, especially after the September 2026 change.
Compare raw HTML with the rendered page. Make sure prices, product names and key claims exist before JavaScript runs.
Decide on Perplexity-User separately. If you need to restrict it, do it at the network layer and accept that you may also block real users' requests.

Frequently asked questions

Is PerplexityBot used to train AI models?

Perplexity's crawler documentation says PerplexityBot is used for search indexing and is not used to crawl content for AI foundation models. That makes it different from training crawlers such as GPTBot or ClaudeBot. Blocking PerplexityBot removes your pages from Perplexity's search index rather than from a training set, so think of it as a search control.

Can I block Perplexity-User with robots.txt?

Not reliably. Perplexity states that Perplexity-User generally ignores robots.txt rules, because a user's request triggers the fetch. If you need to restrict it, the control sits at your firewall or CDN. Be careful: rules strict enough to stop it may also affect legitimate traffic, and they do nothing to PerplexityBot's index-time crawling unless you target that agent too.

Does being in Google or Bing put me in Perplexity?

No. Perplexity builds its own index with PerplexityBot, so a strong Google or Bing presence does not transfer automatically. You can be fully indexed in both and still be missing from Perplexity if your robots.txt or firewall turns PerplexityBot away. Check each engine's access on its own terms.

If PerplexityBot can reach my site, will Perplexity cite me?

Not necessarily. Access makes your pages eligible for the index. Perplexity then decides per query which pages to retrieve and which to cite, and no publisher controls that step. What you can control is removing blocks and making sure the facts that matter are in HTML a non-rendering crawler can read.

How do I know if my firewall blocks PerplexityBot?

Request a real content page using PerplexityBot's user-agent and inspect the response. A 200 with your actual page text is a pass. A 403, a 429, or an HTML challenge or CAPTCHA page is a block. Also confirm robots.txt itself is reachable, and check your server or CDN logs for the status codes PerplexityBot receives.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.