Title: Lab 11 — Building a Merge-Blocking DeepEval Pipeline in GitHub Actions
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 11 — Wiring Evaluation into CI/CD Quality Gates

🛠️ Lab 11 — Building a Merge-Blocking DeepEval Pipeline in GitHub Actions (50 pts)

Goal: Write DeepEval unit tests against a small golden dataset, wire them into a GitHub Actions workflow that runs on every push/PR, watch the pipeline pass on good code, then deliberately break a prompt and watch the same pipeline fail and block the merge — all for free using GitHub's free tier and open-source DeepEval. Steps:

1. Create a free GitHub account if you don't already have one, and create a new public repository (e.g., aiinfra301-lab11).
2. On your local machine, set up a Python virtual environment and install DeepEval: pip install deepeval pytest.
3. Create a golden dataset of 5–8 small test cases in a Python file (golden_dataset.py), each with an input (a realistic user question for a simple RAG app of your choice — reuse your AIINFRA 300 project or a small toy knowledge base), the actual_output your (simple, scripted) RAG pipeline produces, and the retrieval_context it retrieved.
4. Write test_rag.py with a single test function decorated with @pytest.mark.parametrize("case", golden_dataset). Inside the function, instantiate a FaithfulnessMetric(threshold=0.7) (and optionally AnswerRelevancyMetric), build an LLMTestCase from the parametrized case, and call assert_test(test_case, [metric]).
5. Set your judge model's API key as a local environment variable and run deepeval test run test_rag.py locally. Confirm all cases pass and screenshot the output.
6. In your repo, add .github/workflows/eval.yml: a workflow triggered on: [push, pull_request] that checks out the code, sets up Python, runs pip install -r requirements.txt, and runs deepeval test run test_rag.py. Add your judge API key as a GitHub Secret (Settings → Secrets and variables → Actions) and reference it as an environment variable in the workflow step — never hardcode it.
7. Push your code, open a pull request, and confirm in the "Checks" tab that the workflow runs and passes, turning the PR green.
8. Now deliberately introduce a regression: edit your RAG pipeline's prompt template to instruct the model to ignore the retrieved context (or otherwise produce an unfaithful answer), commit, and push to a new branch/PR.
9. Confirm the GitHub Actions check now fails on this PR — screenshot the red "X" and the failing deepeval output showing which metric/threshold was violated.
10. Revert the regression in a third commit/PR and confirm the check goes green again.

Deliverables: Submit a short write-up (1–2 pages) with: (a) your golden_dataset.py and test_rag.py files, (b) your .github/workflows/eval.yml file, (c) the link to your public GitHub repo, and (d) three screenshots — passing workflow, failing workflow (with the regression), and the workflow passing again after the revert — with 2–3 sentences per screenshot explaining what DeepEval flagged and why.