How Perplexity picks the sources it shows
Perplexity does not have one crawler, it has two, and they follow different rules. PerplexityBot builds the search index and is documented as not crawling content for foundation models, while Perplexity-User fetches a page because someone asked a question and generally ignores robots.txt. Which one reaches you decides whether you are indexed or merely quoted once.
Two crawlers, two contracts
Most site owners treat AI crawlers as one category and write a single rule for them. Perplexity's own documentation splits the job in two, and the split matters more than the names.
PerplexityBot is described as designed to surface and link websites in search results on Perplexity, and expressly not used to crawl content for AI foundation models. That is the agent that decides whether your pages are in the index at all. Blocking it removes you from the pool that can be shown, which is a decision worth making deliberately rather than by copying a robots.txt from a forum post.
Perplexity-User is the other half. It supports user actions: someone asks a question, the system visits a page to answer accurately and links to it. Perplexity states plainly that since a user requested the fetch, this fetcher generally ignores robots.txt rules. A blanket disallow therefore does not produce the silence people expect.
The numbers that make the distinction practical
2 documented agents, 2 published IP range files, and 1 of the two honours robots.txt. That is the whole contract, and it fits in three lines of a configuration file.
The commercial side is worth the same amount of attention. In a measured account covering 1 October 2024 to 30 April 2025, traffic from Perplexity converted at 10.5 percent against 1.76 percent for organic search, with ChatGPT at 15.9 percent, Claude at 5 percent and Gemini at 3 percent. The volume behind those rates is small, which is exactly why the rates are the interesting part: a handful of visits that already know what they want.
On the demand side, 45 percent of consumers used an AI tool to find a local business in 2026 against 6 percent a year earlier, and they consult an average of six review sites. A source that appears in several of those six is the one an assistant can safely repeat.
Our own measurement of 187 live hosts in this network found 25 with no robots.txt at all, 6 carrying an llms.txt and 0 blocking an AI crawler. The typical site has no position on any of this, which means the default is being indexed.
| Agent | Purpose | robots.txt |
|---|---|---|
| PerplexityBot | Surfaces and links sites in search results, not model training | Honoured, allowing it is recommended |
| Perplexity-User | Visits a page to answer a live user question | Generally ignored, because a user requested the fetch |
What to do with the two rules
Decide once, in writing, whether you want to be in the index. If the answer is yes, allow PerplexityBot explicitly rather than relying on a wildcard someone may tighten later. If the answer is no, understand that Perplexity-User will still fetch the page when a person asks about you, and plan for the page being read rather than for it being invisible.
Then verify instead of guessing. Both agents publish IP ranges as JSON, so a line in your log claiming to be PerplexityBot can be checked against the published list. Fake agent strings are cheap; matching IP ranges are not.
Questions and answers
Does blocking PerplexityBot keep me out of answers?
Is my content used to train a model?
How do I confirm a visit was genuine?
Is the traffic worth anything?
Sources
- Perplexity, crawler documentation Both agents, their purposes and the robots.txt behaviour
- Seer Interactive, AI traffic conversion study Conversion rates by source for 1 October 2024 to 30 April 2025
- BrightLocal, Local Consumer Review Survey 2026 AI use for local recommendations and the average of six review sites
