Title: Quiz 11 — Wiring Evaluation into CI/CD Quality Gates
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 11 — Wiring Evaluation into CI/CD Quality Gates

Q: Wiring deepeval test run into a GitHub Actions workflow means every pull request automatically runs the eval suite, and a metric falling below its threshold can fail the build and block the merge.
- True
- False
A: True

Q: In a layered LLM CI/CD stack, which tool is best described as validating retriever quality (e.g., context precision and recall)?
- Promptfoo
- RAGAS
- DeepEval
- GitHub Actions
A: RAGAS

Q: Name the three properties a per-PR CI eval check ideally has, of which you can typically only fully satisfy two at once.
Feedback: Sample/expected answer: Cheap, fast, and statistically significant.

Q: What is the practical resolution to the "cheap, fast, statistically significant — pick two" tension in CI eval design?
- Only run evals before major releases
- Use tiered gates: a fast, cheap smoke-test suite per PR, and a fuller statistically rigorous suite run nightly
- Always run the full golden dataset with 20 judge passes on every PR
- Remove evaluation from CI entirely and rely on manual review
A: Use tiered gates: a fast, cheap smoke-test suite per PR, and a fuller statistically rigorous suite run nightly

Q: Because LLM outputs are non-deterministic, a single pass/fail eval run can be flaky; setting the judge model's temperature to 0 and aggregating scores across multiple runs helps produce a more reliable gating decision.
- True
- False
A: True

Q: In the DeepEval pytest integration pattern, which function is called inside a parametrized test function to enforce a metric threshold against a test case?
- `evaluate()`
- `assert_test()`
- `run_metric()`
- `pytest.fail()`
A: `assert_test()`

Q: Why should a judge model's API key be stored in GitHub Secrets rather than hardcoded in the workflow YAML file?
Feedback: Sample/expected answer: So the credential isn't exposed in the repo's source/history or workflow logs; GitHub Secrets stores it securely and injects it as an environment variable at runtime, keeping it out of version control.

Q: Why is "a passing offline eval suite means the change is safe to ship" considered a misconception?
- Offline evals are always more accurate than online monitoring
- Golden datasets can go stale and stop representing real production traffic, creating false confidence unless they're versioned, refreshed, and supplemented with live-traffic sampling
- Offline evals cannot be automated in CI/CD
- DeepEval does not support offline evaluation
A: Golden datasets can go stale and stop representing real production traffic, creating false confidence unless they're versioned, refreshed, and supplemented with live-traffic sampling