Title: Lab 6 — Breaking and Fixing Retrieval with Hybrid Search and RRF
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 06 — Hybrid Search: Combining Dense Retrieval with BM25

🛠️ Lab 6 — Breaking and Fixing Retrieval with Hybrid Search and RRF (50 pts)

Goal: Use a free, local Weaviate instance to observe pure-dense retrieval fail on exact-match queries and watch hybrid search recover them by sweeping the alpha parameter. Then implement Reciprocal Rank Fusion from scratch in Python so the mechanics behind the black box are fully visible. Steps:

1. Stand up Weaviate locally (free, no cloud account needed). Install Docker Desktop (free) if you don't already have it, then run Weaviate via docker compose using the official Weaviate Docker image with a local embedded/free embedding module enabled (e.g., text2vec-transformers running locally, or bring your own vectors from a free sentence-transformers model). Confirm the instance is reachable at localhost:8080.
2. Load a small, deliberately tricky test corpus. Create 20–30 short documents covering a mix of: (a) documents containing exact product IDs, SKUs, or rare technical terms (e.g., QK-9931X, ERR_0x8007042B), and (b) documents that only make sense via paraphrase (no literal keyword overlap with likely queries). Load them into a Weaviate collection configured for hybrid search (vector + BM25 keyword index on the same collection).
3. Design "breaking" queries. Write at least 5 test queries specifically engineered to break pure-dense retrieval: literal product IDs, rare technical terms, and exact error codes that appear in your corpus.
4. Sweep the alpha parameter. Run each breaking query against Weaviate's hybrid search endpoint at alpha = 1 (pure vector), alpha = 0.5 (balanced), and alpha = 0 (pure BM25). Record the rank of the correct document at each setting in a table.
5. Observe and record the recovery. For each query, note where pure-dense retrieval (alpha = 1) fails to surface the correct document in the top 3, and where hybrid settings (alpha = 0 or 0.5) recover it. Write 2–3 sentences per query explaining why the failure occurred, referencing the lecture's explanation of embedding models diluting rare tokens.
6. Implement RRF from scratch in Python. Using the free, local rank_bm25 package (BM25Okapi) as your lexical ranker and a free sentence-transformers model (e.g., all-MiniLM-L6-v2) as your dense retriever, run both over a small free BEIR dataset (e.g., scifact or nfcorpus, downloadable via the beir Python package). Produce two ranked lists per query, then fuse them yourself using score(d) = Σ 1 / (k + rank(d)) with k = 60. Do not use any library's built-in fusion — write the loop yourself.
7. Validate your from-scratch fusion. Compare your manually fused top-5 results against Weaviate's built-in hybrid/RRF output (or against a naive score-averaging baseline you also implement) on the same 3–5 queries, and briefly note where they agree or diverge.
8. Write a short reflection (200–300 words) summarizing what alpha value and RRF k you'd choose for a corpus like the one you built, and why.

Deliverables: Submit your Python notebook/script (RRF implementation + BEIR evaluation), your alpha-sweep results table with the 5+ breaking queries, and your written reflection, either as an uploaded file or pasted directly into the text-entry submission box.