Title: Lecture 6 — Why Hybrid Search Wins: Fusing BM25 and Dense Retrieval with RRF
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 06 — Hybrid Search: Combining Dense Retrieval with BM25

📖 Lecture — Why Hybrid Search Wins: Fusing BM25 and Dense Retrieval with RRF

By now you've built RAG pipelines with dense (embedding-based) retrieval, and it probably worked well on conceptual, paraphrased queries. This week we confront a blind spot you may have already hit: dense retrieval quietly fails on exact-match queries — product codes, named entities, error messages, rare technical terms — precisely where users expect perfect recall. The fix isn't a smarter embedding model; it's combining dense retrieval with the "old-fashioned" lexical method it supposedly replaced: BM25. Two retrievers, two blind spots. BM25 is a lexical scoring function: it rewards documents sharing exact terms with the query, weighted by how rare those terms are corpus-wide (inverse document frequency) and how densely they appear (term frequency, length-normalized). This makes BM25 excellent at exact-term matches — a SKU like TX-4471-B, a rare drug name, an error code — because such terms are rare enough that any matching document scores very highly. But BM25 has no notion of meaning: if a user asks "how do I get my money back" and your docs say "refund policy," BM25 may not connect the two without exact word overlap. Dense retrieval is the mirror image: it excels at paraphrase by comparing meaning in vector space, but underweights rare, specific tokens — an embedding model has seen "refund" far more often than TX-4471-B, so the vector space doesn't reliably preserve rare identifiers. These are complementary retrievers with opposite blind spots, not two flavors of the same tool.

 | BM25 (lexical) | Dense retrieval (vector)
Strength | Exact terms, SKUs, IDs, rare technical terms | Paraphrase, conceptual/semantic queries
Weakness | Synonyms, paraphrase, no notion of "meaning" | Rare/specific tokens diluted in embedding space
Score range | Unbounded (e.g., roughly 0–15+) | Bounded, e.g., cosine similarity –1 to 1 (often 0.6–0.95)
Failure mode | Misses "get my money back" → "refund" | Misses exact match on TX-4471-B

Hybrid search fuses both — and the numbers back it up. Cited studies report hybrid search delivering a 26–31% improvement over dense-only retrieval, with hybrid consistently posting higher NDCG (Normalized Discounted Cumulative Gain, a standard ranked-retrieval metric) than either method alone. That's the empirical case for treating hybrid as the default architecture for production RAG, not an optional add-on. You can't just average the scores. The obvious approach — normalize BM25 and cosine similarity to \[0, 1\] and add them — seems reasonable but breaks in practice. BM25 scores are unbounded and corpus-dependent, while cosine similarity is bounded but compressed into a narrow range. Naive min-max normalization on these incompatible distributions lets outlier BM25 scores dominate the merged ranking, or inflates a tight cluster of cosine scores into false separation. The distributions simply aren't comparable, however you rescale them. Reciprocal Rank Fusion (RRF) sidesteps the problem by ignoring raw scores and fusing on rank position instead: score(d) = Σ 1 / (k + rank(d)) ...summed across each ranked result list a document d appears in. The constant k defaults to 60 — found empirically on TREC data — though k anywhere in \[40, 80\] performs comparably, so it's not a knob to obsess over. Because RRF only looks at "this document ranked 3rd in the BM25 list and 7th in the dense list," it never reconciles an unbounded lexical score with a bounded cosine score. Rank position is a common currency raw scores are not. This is why RRF isn't one vendor's proprietary hack — it's the default hybrid fusion method across OpenSearch, Elasticsearch, Azure AI Search, Weaviate, and Qdrant. Five competing systems converging on the same mechanism signals it solves a fundamental problem correctly. Implementing it in Qdrant. A common pattern stores both a dense_vector and a bm25_sparse_vector on the same point within one collection — no separate index to keep in sync. At query time, you call the Query API with a prefetch parameter (running both sub-queries) plus a fusion parameter set to RRF, returning the fused, re-ranked list in one round trip. You'll practice this in the lab, alongside a from-scratch RRF implementation so the mechanics are never a black box. Correcting two misconceptions. First: normalizing and adding scores is not a safe substitute for RRF — the distributions are structurally incompatible, and naive normalization lets outliers distort rankings, exactly what RRF avoids. Second: dense retrieval has not made BM25 obsolete — BM25 still wins on exact-match queries dense retrieval underweights, which is precisely why hybrid search exists and beats either method alone.