Title: Discussion 11 — Designing Tiered Quality Gates
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 11 — Wiring Evaluation into CI/CD Quality Gates

💬 Discussion 11 — Designing Tiered Quality Gates (20 pts)

Imagine your team ships a RAG feature to production every week, with several pull requests merging per day. You know from this week's lecture that a per-PR CI check must be cheap, fast, and statistically significant — and that you can only fully satisfy two of those three at once. In your initial post, design a concrete tiered gate structure for this team:

- What runs on every PR (which metrics, how many test cases, how many judge runs per case, and why you kept it that small)?
- What runs nightly or on a schedule instead (full golden dataset? multiple judge passes with confidence intervals? live-traffic sampling?), and what happens when it fails (who gets paged, does it block the next release)?
- Which one of "cheap," "fast," or "statistically significant" are you most willing to sacrifice at the per-PR layer, and why is that the right tradeoff for a fast-shipping team?

Then respond to at least two classmates: challenge or build on their tier boundaries — would their per-PR suite actually catch a real regression, or is it too thin? Would their nightly suite realistically get looked at, or would it become noise nobody reads?