Before an AI search agent can cite a page, that page usually has to survive an earlier and less visible competition: first-stage retrieval. From an enormous web corpus, a retriever must identify a relatively small pool of candidate documents that downstream ranking and reasoning systems are allowed to inspect. A relevant page that never enters that pool cannot become evidence later, no matter how useful it might have been.
Perplexity has now published a benchmark designed specifically to measure that gate. Its September 9 research release, Q2D-Web, contains approximately 190 million web documents and 69,721 agent-reformulated queries in ten languages, with three separate sets of relevance judgments intended to evaluate large-scale first-stage retrievers under conditions closer to agentic web search.
The benchmark does not measure the entire Perplexity answer engine, and it does not prove which pages will ultimately receive citations. Downstream reranking, evidence selection and generation still matter. What Q2D-Web measures is earlier: whether a retrieval model can get relevant documents into the candidate set in the first place.
The first retriever defines what the rest of the agent can see
Perplexity describes a production-scale retrieval system as searching a corpus containing billions of web pages and returning thousands of candidate documents for a query. Those candidates determine which sources downstream systems can inspect.
This creates a hard information boundary. A reranker can reorder candidates, and a language model can decide which retrieved evidence deserves to support an answer, but neither can recover a useful page that the first-stage retriever never surfaced.
For publishers thinking about AI visibility, that means citation is not a single ranking event. It is the result of a funnel. The page must first become retrievable for the agent’s reformulated query, survive candidate generation, remain competitive through later ranking and finally be selected as useful evidence.
Q2D-Web isolates the earliest major gate in that process.
Q2D-Web contains 69,721 agent-generated search queries
The query set is notable because it does not consist primarily of short, human-written benchmark questions. Perplexity built it from 23,000 production searches collected across nine months.
A production search can contain multiple calls to web-search tools. Each tool call can contain one or more queries, and Perplexity says those queries are reformulated from the user’s request, prior conversation and, when applicable, documents already retrieved earlier in the same research process.
Each search contains a main query representing the core information need and can include supporting queries exploring alternative formulations, background information or related entities. Q2D-Web evaluates those queries separately because different formulations can require different relevant documents even when they originate from the same user task.
The final benchmark contains 69,721 such agent-reformulated queries rather than exactly 70,000; Perplexity rounds the figure to 70 thousand in its research index and headline description.
The queries cover ten languages and many information domains
English accounts for 65.8% of Q2D-Web queries. The remaining queries include Spanish, Russian, German, French, Portuguese, Italian, Korean, Japanese and Chinese.
Perplexity also sampled across a broad range of topics, including consumer goods, programming, law, health, economics, science, education, technology, finance, travel, entertainment, politics, local information and news.
This diversity matters because retrieval performance can vary substantially across languages and domains. A model that performs well on English technical questions may behave differently on local information or multilingual consumer research.
The benchmark therefore attempts to reflect the heterogeneous query distribution an agentic search product encounters rather than reducing web retrieval to one narrow academic domain.
Perplexity says the production queries are filtered for privacy
Because the benchmark originates from real searches, data governance is part of its construction. Perplexity says the included queries come from users who consented to having their data used and that its perplexity-ai/pplx-pii-masking system is used to exclude queries containing personally identifiable information.
The company also removes exact duplicates, very short queries and queries containing explicit search operators such as site: or quoted phrases.
Perplexity describes the resulting production traffic as PII-free. That is the company’s characterization of its filtered dataset; it should not be interpreted as independent verification that an automated privacy filter can never miss personal information.
Still, the filtering process is material because Q2D-Web is explicitly designed to capture realistic agent search behavior without simply publishing raw production queries.
The 190-million-document corpus is intentionally difficult
Q2D-Web does not use a random sample of the web. For every benchmark query, Perplexity retrieves the top 5,000 documents using a production retrieval system and takes the union of those result sets.
The corpus is then deduplicated using MinHash-LSH. Perplexity groups documents when their token 5-gram sets have a Jaccard similarity of at least 0.975.
This process produces roughly 190 million documents, each of which was considered a plausible result for at least one benchmark query. That design intentionally creates hard negatives: pages that look semantically relevant but may miss the specific entity, date, version, quantity or detail required by the query.
A retriever therefore cannot succeed simply by finding documents that are vaguely on-topic. It has to distinguish genuinely useful evidence from large numbers of plausible distractors.
Why a smaller corpus can make retrieval look artificially good
Perplexity argues that benchmark corpus size matters because removing hard negatives can inflate recall. If a benchmark preserves known relevant documents while discarding many realistic distractors, the retrieval problem becomes easier than production web search.
That can produce a misleading impression of how well an embedding model will perform when deployed against hundreds of millions or billions of pages.
Q2D-Web is designed to push on three dimensions simultaneously: corpus size, number of judged queries and depth of relevance judgments per query.
Perplexity contrasts this with benchmarks that may offer web-scale corpora but relatively few judged queries, or many queries against much smaller document collections.
The benchmark uses three different definitions of relevance
One of the harder problems in retrieval evaluation is knowing which documents are truly relevant. At web scale, manually judging every query-document pair is impossible.
Q2D-Web therefore provides three relevance sets rather than treating one labeling pipeline as ground truth. The first is Citation: a document is considered relevant when an agent actually cited it in an answer.
This signal is close to downstream behavior because the agent selected the page as evidence for a claim. But Perplexity notes that citation has high precision and low recall. Once an agent has enough evidence, it has little reason to cite every other page that would also have been relevant.
A page can therefore be highly useful and still lack a Citation label simply because another source already supplied sufficient support.
Production web rankings provide a broader relevance set
The second relevance set is Web Ranking. Perplexity takes up to 50 documents per query, averaging 43.1, from an internal retrieval stack that combines BM25 and dense retrieval at the first stage and then applies cross-encoder reranking.
This expands the relevant set beyond pages the agent actually cited. It captures documents that Perplexity’s production ranking system considered useful even when they were not selected as explicit evidence in the final answer.
The advantage is broader coverage. The limitation is that the labels inherit biases from the production retrieval and ranking system that generated them.
This is one reason Perplexity does not rely on Web Ranking alone.
LLM judgments try to reduce false negatives further
The third construction, Combined + LLM-Judged, takes the union of Citation and Web Ranking labels and adds judgments for previously unjudged candidates.
Perplexity pools results from BM25, ColBERTv2 and seven dense retrievers released before January 1, 2025. It merges those rankings using reciprocal rank fusion, selects the top 500 unjudged documents and applies a strict binary relevance judge using DeepSeek-V4-Flash.
The purpose is to identify relevant documents that the earlier labeling systems missed. Perplexity reports an average of 99.6 positive relevance judgments per query in the combined set.
Using several judgment sources does not create perfect ground truth, but it reduces dependence on any single pipeline and makes it less likely that a genuinely relevant document will be incorrectly counted as a retrieval failure merely because nobody had labeled it before.
Citations become a training signal for evaluating retrieval
The Citation relevance set creates an interesting feedback loop between agent behavior and retrieval evaluation. Pages that agents actually selected as evidence become one way to test whether first-stage retrievers can recover documents with demonstrated downstream usefulness.
This does not mean every cited page is objectively the best source or that every uncited page is irrelevant. Perplexity explicitly recognizes the incompleteness of citation labels.
But it gives Q2D-Web a signal that traditional information-retrieval datasets often lack: evidence that a document was not merely topically related, but was used by an agent to support an answer.
For GEO analysis, that is particularly interesting because it connects the first retrieval gate to the eventual citation layer without pretending the two are identical.
Recall@1000 measures whether evidence enters the large candidate pool
Q2D-Web reports several retrieval metrics, including Recall@1000, Recall@100 and nDCG@10. Each answers a different question about the evidence funnel.
Recall@1000 asks whether relevant documents appear somewhere in a relatively broad candidate set. For a first-stage retriever, this is critical because downstream systems cannot use evidence that is missing entirely.
Recall@100 creates a narrower test, while nDCG@10 gives greater weight to placing relevant documents near the top of the ranking.
A model can therefore perform differently depending on whether the goal is broad evidence coverage or highly precise ordering near the top.
No model wins every relevance definition
Perplexity’s reported results illustrate why retrieval quality cannot be reduced to one leaderboard number. On full-corpus Recall@1000, its pplx-embed-v1-4b model leads the Web Ranking and Combined judgment sets, with scores of 65.73 and 69.11 respectively, while Nemotron-3-Embed-8B leads the Citation set at 61.68.
The ordering changes on other metrics. Perplexity reports that Nemotron-3-Embed-8B is ahead of pplx-embed-v1-4b on Combined Recall@100 and Combined nDCG@10.
This means the “best” retriever depends partly on what the evaluation values: recovering many relevant documents in a large candidate pool, finding citation-associated documents or ranking the strongest evidence very high.
For agentic RAG, those objectives can matter differently depending on the downstream reranker and how many documents the agent is able to inspect.
Larger embedding models generally recover more evidence within a family
Perplexity also reports a consistent size effect within several embedding-model families on Combined Recall@1000. Qwen3-Embedding rises from 57.89 for the 0.6B model to 64.53 for the 8B version.
Nemotron-3-Embed increases from 61.68 at 1B parameters to 68.58 at 8B, while Perplexity’s own pplx-embed-v1 family rises from 67.02 at 0.6B to 69.11 at 4B.
Those results suggest that more capable embedding models can improve the probability that relevant documents survive first-stage retrieval, although model size is only one variable and does not guarantee leadership across every metric.
For production systems, the quality gain also has to be weighed against indexing and inference cost.
Evaluating 190 million documents is computationally expensive
Perplexity says one full Q2D-Web evaluation with pplx-embed-v1-4b requires 4,608 H200 GPU hours. Even a much smaller model such as EmbeddingGemma-300M requires nearly 200 H200 GPU hours.
That cost makes repeated full-corpus evaluation impractical for many research teams. Perplexity therefore also constructed a smaller sampled corpus intended to preserve the behavior of the full benchmark closely enough for more practical experimentation.
The company performed ablations on sampling strategies because naive corpus reduction can remove the hard negatives that make large-scale retrieval difficult and artificially boost measured recall.
This is another reminder that web-search evaluation is partly an infrastructure problem. Realistic retrieval benchmarks become expensive simply because the candidate universe is enormous.
Perplexity acknowledges a potential home-field advantage
Perplexity evaluates its own embedding models using the same indexing, retrieval and scoring pipeline applied to other publicly released models on the leaderboard. It also says the benchmark queries and corpus were excluded from its models’ training data.
However, the company explicitly notes that Q2D-Web originates from Perplexity production traffic. Its models may therefore benefit from an in-distribution advantage even without direct training on the evaluation set.
That caveat is important when interpreting the leaderboard. A model developed around similar query distributions may naturally align better with the benchmark than one optimized for a different retrieval environment.
The public leaderboard is useful comparative evidence, but it should not be treated as a universal ranking of embedding models for every search product.
For publishers, the benchmark exposes the citation gate before the citation
Q2D-Web provides a useful mental model for AI visibility. Publishers often look at the final answer and ask why a competitor received a citation while their own page did not.
That question may be too late in the pipeline. If the page failed first-stage retrieval for the agent’s reformulated query, it never entered the evidence pool that later systems could rank or cite.
This shifts part of GEO analysis toward retrievability. A page must communicate the relevant entities, concepts, relationships and specifics strongly enough that embedding or hybrid retrieval systems recognize it as a plausible answer to the agent’s query.
That does not reduce AI visibility to embedding similarity. The retrieved candidate still has to survive later ranking and provide evidence the model considers useful. But without candidate generation, none of those later opportunities exist.
Agent reformulation means publishers are optimizing for queries they may never see
Another important implication comes from how Q2D-Web queries are created. They are not necessarily copies of the user’s original wording.
The agent can reformulate the request using conversation context and previously retrieved information, then issue supporting searches for narrower facts or related entities.
A publisher may therefore be retrieved for an intermediate machine-generated query that no human typed directly. The language of AI visibility can diverge from the language of traditional keyword research.
Pages that clearly express facts, entities, attributes and relationships may be easier for retrievers to match across these reformulations than pages that depend heavily on one exact keyword phrase.
The benchmark does not tell publishers exactly how Perplexity ranks production results
Q2D-Web is a benchmark for evaluating embedding retrievers, not a complete disclosure of Perplexity’s live search ranking algorithm. Its corpus is constructed from production retrieval candidates, but the leaderboard uses a consistent evaluation pipeline so publicly released models can be compared.
Production Perplexity can combine multiple retrieval systems, rerankers, freshness signals and other proprietary components not captured by a single benchmark score.
Publishers should therefore avoid turning Q2D-Web into a simplistic recipe such as “optimize for the leaderboard winner.” The benchmark reveals the importance and difficulty of first-stage retrieval; it does not expose every feature used by Perplexity Search.
Its strongest SEO value is conceptual and methodological: it shows that evidence visibility begins well before the final answer is generated.
AI citation measurement needs to study the whole retrieval funnel
Q2D-Web also suggests a better framework for AI-search analytics. Measuring only final citations collapses several distinct failure modes into one outcome.
A page may fail because it was never retrieved, because it ranked too low after retrieval, because another source offered stronger evidence or because the answer generator decided the claim did not require that citation.
Those are different problems and require different remedies. First-stage retrieval is fundamentally about evidence-pool admission; later stages are about prioritization and use.
As AI visibility tooling matures, separating these layers could become more valuable than simply counting how often a domain appears in final responses.
Perplexity has benchmarked the gate that decides what an agent is allowed to know
Q2D-Web’s scale—69,721 agent-reformulated queries, roughly 190 million web documents and three complementary relevance sets—makes it a substantial attempt to evaluate the first stage of agentic web retrieval under realistic conditions.
The benchmark does not solve citation measurement, and it does not establish a deterministic path from retrieval to citation. What it does is quantify an earlier prerequisite: can the retriever surface the relevant source from a huge collection of plausible alternatives?
For publishers, that is the gate before every downstream ranking decision. A page can be authoritative, well written and perfectly suited to support an answer, but an agent cannot cite evidence it never receives.
Perplexity’s benchmark makes that hidden stage more measurable. In the emerging AI-search funnel, being citation-worthy is not enough. The first retriever has to let the page into the room.