Title: Lab 7 — Retrieve, Rerank, and Rewrite: Measuring the Lift
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 07 — Cross-Encoder Rerankers and Query Rewriting/Expansion

🛠️ Lab 7 — Retrieve, Rerank, and Rewrite: Measuring the Lift (50 pts)

Goal: Build a two-stage retrieve-and-rerank pipeline on a free benchmark dataset, measure the precision lift and latency cost of reranking, then extend the pipeline with HyDE-style query expansion — confirming that the original query, not the rewritten one, is what gets passed to the reranker. Steps:

1. Set up your environment. Using free/local tools only, install sentence-transformers (which includes both bi-encoder and cross-encoder support) via pip. No paid API keys are required for this lab.
2. Load a free BEIR dataset. Download the SciFact dataset (a small, free BEIR benchmark of scientific claims and supporting documents) using the beir Python package or the dataset's public download link. Load its corpus, queries, and relevance judgments (qrels).
3. Stage 1 — bi-encoder retrieval. Embed the corpus and queries with a small open bi-encoder (e.g., sentence-transformers/all-MiniLM-L6-v2). For a sample of at least 20 queries, retrieve the top-50 candidate documents per query using cosine similarity.
4. Measure baseline precision. Using the provided qrels, compute Precision@5 (or nDCG@5) for the bi-encoder-only results on your sample queries. Record this number — it's your baseline.
5. Stage 2 — cross-encoder reranking. Load cross-encoder/ms-marco-MiniLM-L-6-v2. For each query, score all 50 candidates jointly with the query text, then re-sort by the cross-encoder score and keep the top 5.
6. Measure reranked precision and latency. Recompute Precision@5 (or nDCG@5) on the reranked top-5 and compare to your baseline. Separately, time the reranking step per query (wall-clock, CPU) across your sample and report the average added latency in milliseconds.
7. Add HyDE-style query expansion. In front of retrieval, use a local/free LLM (e.g., a small Ollama model, or any free-tier/local model you already have configured from AIINFRA 300) to generate a hypothetical answer for each query. Embed that hypothetical answer and use it — instead of the raw query embedding — for Stage 1 retrieval.
8. Verify the original-query rule. In your pipeline code, confirm explicitly (with a comment and/or an assertion) that Stage 2 reranking scores each candidate against the original user query text, never the HyDE-generated hypothetical answer or any rewritten variant.
9. Re-measure with HyDE in front. Repeat your Precision@5/nDCG@5 measurement with HyDE-based retrieval feeding into the same cross-encoder reranker. Compare all three conditions: bi-encoder only, bi-encoder + reranker, bi-encoder(HyDE) + reranker.
10. Write up your findings. In 1-2 pages, present your three-condition comparison table (precision metric + added latency for reranking), and answer: was the precision lift from reranking worth the latency cost? Did HyDE improve results further, and was that gain worth its own added LLM-call latency?

Deliverables: Your pipeline code (Python script or notebook) covering all three conditions, plus a 1-2 page write-up (PDF or text) with your comparison table and analysis, submitted as an upload or pasted as text entry.