Title: Lab 10 — Catching Failures with DeepEval Metrics
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 10 — Deep Evaluation and Failure Analysis with DeepEval

🛠️ Lab 10 — Catching Failures with DeepEval Metrics (50 pts)

Goal: Install DeepEval, wire it up to your own RAG pipeline (or the sample pipeline from AIINFRA 300 if you don't have one handy), and prove that DeepEval's RAG metrics actually catch failures — including ones you deliberately plant — then write a custom G-Eval metric for a quality dimension the built-in metrics don't cover. Steps:

1. Create a clean virtual environment for this lab: python -m venv deepeval-lab then activate it (source deepeval-lab/bin/activate on macOS/Linux, deepeval-lab\Scripts\activate on Windows).
2. Install DeepEval: pip install -U deepeval. Confirm the install with deepeval --version.
3. Configure a local judge model so evaluation runs entirely free and local: install and run Ollama, pull a small instruction-tuned model (e.g. ollama pull llama3.1), then point DeepEval at it following the local-model setup in the DeepEval docs (see Resources). If you'd rather use a hosted API key you already have from a prior course, that's fine too — just document which judge model you used.
4. In a new file test_rag_pipeline.py, import LLMTestCase and the RAG metrics you need: AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric, ContextualRecallMetric.
5. Build 4–6 LLMTestCase objects from your own pipeline's real input/output/retrieval-context, or from the AIINFRA 300 sample pipeline if you don't have your own yet. Each test case needs: the user question (input), your pipeline's answer (actual_output), the chunks your retriever returned (retrieval_context), and — where available — a reference answer (expected_output).
6. Deliberately inject two failing cases so you can confirm the metrics actually work as failure detectors, not just pass everything: (a) a hallucinated-fact case, where you hand-edit actual_output to state a fact not present in retrieval_context; (b) a missing-context case, where you truncate retrieval_context so it's missing information the question needs, then observe what happens to the answer's Contextual Recall.
7. Run the suite with the CLI workflow: deepeval test run test_rag_pipeline.py. Confirm your two planted failures actually fail their respective metrics (Faithfulness for the hallucination, Contextual Recall for the missing-context case) while your healthy cases pass.
8. Drill into the failure reasons: use deepeval test run verbose output (or deepeval inspect for the interactive TUI view) to read each metric's explanation for why it scored a case low. Capture at least one full explanation per planted failure in your submission.
9. Write one custom G-Eval metric for a quality dimension specific to your own project that the built-in metrics don't cover — for example, "does the answer cite a source" or "is the tone appropriately cautious about medical/legal claims." Write a specific criteria string (not a vague one — apply this week's misconception lesson) or explicit evaluation_steps, run it against your test cases, and report the scores.
10. Write a short reflection (150–250 words) covering: which metric caught which planted failure and why, one thing that surprised you about a failure-reason explanation, and how you kept your custom G-Eval criteria specific enough to be reproducible.

Deliverables: Submit test_rag_pipeline.py (or a notebook), the console/TUI output showing your deepeval test run results with both planted failures caught, your custom G-Eval criteria string and its scores, and your written reflection — as an upload or pasted into the text entry box.