Title: Lab 9 — Scoring Your RAG Pipeline with RAGAS
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 09 — RAG Evaluation Foundations with RAGAS

🛠️ Lab 9 — Scoring Your RAG Pipeline with RAGAS (50 pts)

Goal: Install RAGAS, run all four core metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall) against a real RAG pipeline, and use the resulting scores to make and justify one concrete engineering decision about your retriever, chunk size, or prompt. Everything in this lab runs free on Google Colab (or your own machine) using free/local tools — no paid API spend is required. Steps:

1. Open a fresh Google Colab notebook (or local Python environment) and install RAGAS: pip install ragas.
2. Bring or build a RAG pipeline to evaluate. Use the pipeline you built in a prior AIINFRA 300/301 lab if you have one, or stand up a minimal retriever quickly using a free BEIR dataset with real relevance judgments — SciFact or NFCorpus are good choices because they are small and ship with ground-truth relevance labels (see BEIR link in Resources).
3. Build a small evaluation set of 8-15 (question, ground_truth_answer) pairs. If you used SciFact/NFCorpus, you can derive ground-truth answers from the labeled relevant documents; otherwise write your own reference answers for questions your pipeline should be able to answer.
4. Run your pipeline over each question to produce (question, retrieved_contexts, generated_answer) triples. Save these alongside your ground-truth answers — this is the dataset RAGAS will score.
5. Configure a judge LLM via environment variable (an API key for your provider of choice), or use a local/free model if you want to avoid any paid API cost. Follow the RAGAS quickstart pattern (ragas quickstart rag_eval or manual setup) and, if you want to practice the pattern from lecture, configure Claude as the judge LLM using llm_factory instead of the default.
6. Run evaluate() from RAGAS against your dataset using all four core metrics: Faithfulness, Answer Relevancy, Context Precision, Context Recall.
7. Export the results to a table (CSV or a markdown table in your notebook) showing per-question scores and the averages across your whole set.
8. Analyze the pattern in your scores. Identify at least one metric that is noticeably weaker than the others and write a short diagnosis: is the problem in retrieval (precision vs. recall — be specific about which) or in generation (faithfulness vs. relevancy)?
9. Based on your diagnosis, make one concrete change — e.g., adjust chunk size, switch or re-rank your retriever, tighten your generation prompt to stick closer to context — and re-run the evaluation on the same question set.
10. Compare before/after scores and write 150-250 words explaining what you changed, why you expected it to move the specific metric you targeted, and whether it actually did.

Deliverables: Submit your Colab notebook (or .py script) with all cells/output visible, your before/after RAGAS score table, and your 150-250 word written analysis, either as an uploaded file or pasted into the text entry box.