Title: Lecture 11 — From Manual Evals to Automated Quality Gates
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 11 — Wiring Evaluation into CI/CD Quality Gates

📖 Lecture — From Manual Evals to Automated Quality Gates

Up to now you've been running evaluations by hand: write a script, run it against a test set, eyeball the numbers, decide if things look okay. That works while you're the only one touching the prompt. It falls apart the moment a team ships a RAG feature every week, because a workflow that depends on someone remembering to run the eval will eventually run without it — usually right before the release that needed it most. The fix is the one software engineering already applied to unit tests: put the eval suite in CI, and make a failing score block the merge. The merge-blocking gate. The core pattern is deepeval test run wired into GitHub Actions (or GitLab CI): every pull request triggers the workflow, the workflow runs your eval suite against a golden dataset, and if a metric — say, faithfulness — drops below a defined threshold, the build fails and the PR cannot merge. This is no different in spirit from a failing pytest run blocking a merge; the only new wrinkle is that the "test" is an LLM-as-judge score instead of an assertion on a return value. Once this is in place, a prompt change, a retriever swap, or a model upgrade can't silently ship a quality regression — the pipeline catches it before it reaches production. A layered stack, not one tool. A mature 2026 LLM CI/CD pipeline rarely relies on a single evaluation tool, because each tool is good at a different layer of the stack. A practical combination:

Tool | What it validates | Where it runs
RAGAS | Retriever quality (context precision/recall, retrieval relevance) | CI, per-PR or nightly
DeepEval | Granular unit tests (faithfulness, answer relevancy, custom metrics) wired directly into pytest/CI runners | CI, per-PR
Promptfoo | Prompt-level regressions (does a prompt edit break known good/bad cases?) | CI, per-PR

Running these together as layered quality gates means a regression introduced anywhere in the pipeline — bad chunking, a careless prompt tweak, a model swap — gets caught by the tool best suited to notice it, instead of hoping one generic score covers everything. The engineering-judgment tension: pick two of three. Here's the tension you need to internalize, because it's the crux of designing any CI eval strategy: a check that runs on every single PR must be cheap, fast, and statistically significant — and you can only ever fully have two of the three. A single LLM-judge call per test case is fast and cheap but noisy (not statistically significant on its own). Running the judge 20 times per case and aggregating is statistically solid but slow and expensive. Running your entire golden dataset with multiple judge passes on every commit is thorough but neither cheap nor fast enough for a team merging several times a day. The practical resolution is tiered gates: a small, fast smoke-test suite (a handful of high-signal golden cases, single judge pass, temperature 0) runs on every PR and blocks the merge if it fails. A larger, statistically rigorous suite — full golden dataset, multiple judge runs per case, confidence intervals — runs nightly or on a schedule, and its failures page the team or block the next release rather than blocking every individual PR. This is how you get "cheap, fast, and statistically significant" as close to simultaneously as two-out-of-three honestly allows. The Pytest integration pattern. Concretely, DeepEval tests look like ordinary pytest functions. You write a test function decorated with @pytest.mark.parametrize that iterates over your golden test cases (input, expected output, retrieved context), and inside the function body you call assert_test() with the metric(s) you want enforced (e.g., FaithfulnessMetric(threshold=0.8)). Run the whole file with deepeval test run test_rag.py, which is DeepEval's own pytest wrapper — it gives you richer reporting than plain pytest while still behaving like a normal test run. In the GitHub Actions workflow, this becomes a job that checks out the repo, installs Python dependencies (including deepeval), and runs deepeval test run as a step. Any API keys the judge model needs (e.g., an OpenAI key) go into GitHub Secrets, never into the workflow file or the repo. Misconception: "gate on a single run, like a unit test." A regular unit test is deterministic — same input, same output, every time. An LLM call is not: identical inputs can produce different outputs from one run to the next, and an LLM-as-judge score inherits that same variance when it scores those outputs. If you gate a release on one single pass/fail number, you will get flaky builds — a PR that would pass nine times out of ten fails on the tenth, unrelated to any real regression. The correction is to treat this as a statistics problem, not a boolean check: set temperature=0 for the judge model to reduce (not eliminate) run-to-run variance, and aggregate scores across multiple runs with an adequate sample size — using confidence intervals rather than a single number — before you let a score block a release. Misconception: "a passing offline suite means the change is safe to ship." A golden dataset is a snapshot of what your users asked and what "good" looked like at the time you built it. Real traffic drifts — new topics, new phrasing, new edge cases — and a golden dataset that isn't refreshed quietly stops representing production, even while it keeps returning green checkmarks. That's false confidence, not safety. The correction: version your golden datasets alongside your code (so you know exactly which dataset a given commit was tested against), periodically refresh them by sampling real production traffic, and don't stop evaluating at the merge — keep sampling and scoring live traffic after release so drift shows up before it becomes a user-visible failure. Put together: CI/CD quality gates don't replace judgment, they encode it. Tiered gates encode the cheap/fast/significant tradeoff; multi-run aggregation encodes the fact that LLM outputs are random variables, not fixed values; and live-traffic sampling encodes the fact that "passing" is a moving target. Get comfortable with all three, and you're doing production LLMOps rather than one-off scripting.