Title: Lecture 7 — Reranking and Rewriting: Getting the Right Documents to the Top
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 07 — Cross-Encoder Rerankers and Query Rewriting/Expansion

📖 Lecture — Reranking and Rewriting: Getting the Right Documents to the Top

By now your RAG pipeline can retrieve documents fast. This week is about making sure the documents it retrieves are actually the right ones — and that the query it searches with is a good query in the first place.

The two-stage cascade: retrieve broad, rerank narrow

Every bi-encoder retriever you've used so far — the kind that embeds queries and documents into the same vector space and compares them with cosine similarity or dot product — has a fundamental limitation. It encodes the query and each document separately, with no interaction between them, before comparing the resulting vectors. That's exactly what makes bi-encoders fast enough to search millions of documents: you can precompute every document vector offline and only embed the query at request time. But it also means the model never gets to reason jointly about "does this specific document answer this specific question." Nuance gets flattened into a single vector. A cross-encoder fixes that at the cost of speed. It feeds the query and a candidate document together into one transformer pass, lets the attention layers compare them token-by-token, and outputs a single relevance score. This is much more accurate — but it cannot be precomputed, because you don't know the query in advance. That's why the standard production pattern is a cascade, not a replacement:

1. Stage 1 — Retrieve broad (bi-encoder): Your existing vector search pulls back a generous candidate set, say the top 50-100 chunks, cheaply and in milliseconds.
2. Stage 2 — Rerank narrow (cross-encoder): The cross-encoder scores each of those 50-100 candidates jointly with the query and reorders them by true relevance. You keep only the top 3-10 after reranking.

This is why we say "retrieve-broad-then-rerank-narrow," not "rerank everything." A cross-encoder scores one query-document pair per forward pass, with no precomputed document vectors to fall back on — so running it over your entire corpus would be computationally impossible. Reranking only pays off because it only ever touches the small candidate set the first stage already narrowed down. Added latency for a candidate set of \~50 is typically only about 50-300ms on CPU, which is a small price for a meaningful precision jump before your LLM ever sees the context.

Choosing a reranker

Reranker | Type | Approx. latency (50 candidates, CPU) | Notes
cross-encoder/ms-marco-MiniLM-L-6-v2 | Open, self-hosted | \~100-300ms | Fast, small, strong default; trained on MS MARCO (web search Q&A)
BAAI/bge-reranker-v2-m3 | Open, self-hosted | Higher (larger model) | Stronger multilingual quality, more compute
Cohere Rerank | Hosted API | Network-bound, no local compute | No infrastructure to manage; usage-based cost instead of GPU/CPU cost

Pick based on your constraints: if you need to run fully offline or at near-zero marginal cost, start with ms-marco-MiniLM-L-6-v2. If your users query in multiple languages, bge-reranker-v2-m3 is worth the extra compute. If you'd rather not host a model at all, Cohere Rerank trades an API dependency for operational simplicity.

Rewriting the query before you ever retrieve

Reranking fixes ordering problems in what you retrieved. It can't fix a query that was a poor search query to begin with. Two techniques address that upstream: HyDE (Hypothetical Document Embeddings): instead of embedding the user's raw question, you ask an LLM to write a hypothetical answer to it, then embed that hypothetical answer and use it to search. The intuition: answers look more like the documents you're searching (same vocabulary, same style) than questions do, so embedding an answer-shaped piece of text often lands closer to the real answer in vector space. Multi-query expansion: LangChain's MultiQueryRetriever automates having an LLM generate several rewritten variants of the user's query, runs retrieval for each, and fuses/deduplicates the combined results (often with Reciprocal Rank Fusion, RRF). Current guidance is to generate 3-6 rewrites — beyond that, you start flooding the retriever with too many near-duplicate queries and the returns diminish.

What actually still helps, once you have a reranker

A natural question once you've added a cross-encoder is: what else is worth building? Recent evaluation work (arXiv:2606.28367) tested a stack of retrieval-enhancement tricks layered on top of a strong reranker and found a clear ordering: once a strong cross-encoder reranker is in place, query expansion is the one enhancement that still yields a reliable, additional gain. Several other tricks — expanding the candidate pool further, adding a second reranking pass, and similar embellishments — added little to nothing once reranking was already doing its job. The practical takeaway for you as a builder: invest first in getting a reranker into your pipeline, then invest in query expansion (HyDE or multi-query) as your next enhancement, and be skeptical of additional retrieval "tricks" that promise gains beyond that without evidence.

Correcting three common misconceptions

"A cross-encoder reranker is just another embedding-similarity step, so it scales like retrieval." Not so. A cross-encoder scores the query and a document jointly in a single transformer pass, per candidate — there are no precomputed document vectors to reuse. That's precisely why it can only ever run over a small candidate set (retrieve-broad-then-rerank-narrow), never your whole corpus. "A reranker trained on MS MARCO works everywhere, and more query expansion is always better." Cross-encoders trained on general web-search data like MS MARCO can underperform badly out-of-domain — medical, legal, and code corpora all have vocabulary and relevance patterns that differ from web search. Similarly, query expansion has a ceiling: pushing past a reasonable number of rewrites or added terms starts pulling in irrelevant documents rather than helpful ones. "Pass the LLM-rewritten query straight to the reranker, since it's supposedly a better query." The rewritten or expanded query is a tool for retrieval — casting a wider or more targeted net over your document store. But the reranker's job is to judge relevance against what the user actually asked. Always score reranker candidates against the original user query, not the rewritten one, or you risk optimizing relevance toward a question nobody asked.