Rerankers Earn Their Latency, Two-Stage Retrieval Economics

Pull-quote: “The first stage is paid to be fast and forgetful. The second stage is paid to be careful. Most retrieval stacks fail by asking one stage to do both jobs.”

Why this matters

A single-stage retriever has an impossible job description: scan millions of chunks in milliseconds and also judge fine-grained relevance. The architecture makes the compromise explicit. Bi-encoder embeddings compress query and document into separate vectors, so search reduces to fast nearest-neighbor lookup, but the model never reads the query and the passage together. It compares summaries of meaning. That is the price of speed, and no amount of embedding quality removes it.

Two-stage retrieval splits the job along its natural seam. Stage one optimizes recall: cast wide, cheaply, over everything. Stage two optimizes precision: read carefully, expensively, over the shortlist.

The division of labor

The cross-encoder reads query and passage jointly, token by token. That is what restores sensitivity to negation, direction, and specificity, exactly the distinctions bi-encoders blur. The candidate count is the economic control knob: rerank 50 when latency is tight, 200 when recall is precious.

The reranker pays twice

First, answer quality. The ordering that reaches the generator is the ordering a careful model produced, not the ordering a compressed vector guessed.

Second, and less appreciated, the token budget. Teams without a reranker compensate for noisy ranking by stuffing more chunks into the prompt and letting the generator sort it out. The generator is the most expensive component per token in the whole pipeline, and irrelevant passages are not neutral filler; they are distractors that degrade the answer. Sending six well-ordered passages instead of twenty hopeful ones shrinks the prompt, sharpens the output, and frequently reduces end-to-end latency. The reranker spends milliseconds to buy back the generator’s.

   Corpus: millions of chunks
        │   stage one: BM25 + vector ANN, rank fusion
        ▼
   ~200 candidates         recall-oriented, cheap per query
        │   stage two: cross-encoder reads query + passage jointly
        ▼
   5-10 passages           precision-oriented, paid per candidate
        │
        ▼
   Generator prompt        fewer, better passages: smaller, cheaper, sharper

When to skip the second stage

Skip it on strict interactive latency floors where retrieval is ambient to typing. Skip it on tiny, curated corpora where stage one is already precise. Deduplicate before you rerank, because a cross-encoder cannot fix a candidate list that is forty copies of the same boilerplate. And measure: if reranking rarely changes the head of your ranked list on real traffic, pocket the latency and revisit when the corpus grows.

Where this shows up in the field

Safety corpora justify the second stage most clearly, because safety questions hinge on distinctions that similar-sounding narratives blur: which crew action preceded which alert, which system was inoperative rather than degraded. Evidence synthesis raises the stakes further. A marginal passage admitted into a review does not just waste tokens; it dilutes the evidence the conclusion stands on. In both settings the second stage is not overhead. It is the safeguard.

Closing

Two-stage retrieval is not an optimization trick. It is an honest division of labor between a stage built to never miss and a stage built to never guess. Size the first stage for recall, budget the second by candidate count, and let the generator see less and better. The latency you spend in the middle of the pipeline is routinely cheaper than the tokens and the trust you spend at the end of it.