Title: Quiz 10 — Deep Evaluation and Failure Analysis with DeepEval
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 10 — Deep Evaluation and Failure Analysis with DeepEval

Q: In DeepEval, what does a metric return for a given test case?
- A pass/fail boolean only
- A score between 0 and 1
- A raw token count
- A ranked list of alternative outputs
A: A score between 0 and 1

Q: A DeepEval test "passes" only when its metric score meets or exceeds a configurable threshold you set.
- True
- False
A: True

Q: Why do BLEU/ROUGE-style n-gram overlap metrics often score a correct paraphrased RAG answer poorly?
- They require GPU acceleration DeepEval doesn't support
- They only measure surface word/phrase overlap, not semantic meaning
- They can only be run on non-English text
- They ignore the question entirely and only score length
A: They only measure surface word/phrase overlap, not semantic meaning

Q: In G-Eval, what are the two mutually exclusive ways you can supply the evaluation logic, and why can you not use both at once?
Feedback: Sample/expected answer: You supply either a plain-language ****`criteria`**** string (which G-Eval auto-generates chain-of-thought evaluation_steps from) or explicit ****`evaluation_steps`**** yourself — not both, because G-Eval derives one path of reasoning steps and mixing a criteria string with hand-written steps would create conflicting instructions for the judge.

Q: How does G-Eval convert its raw 1-5 LLM judgment into a normalized 0-1 score?
- By dividing the raw score by 5
- By averaging scores across three separate judge models
- By using the underlying token probabilities of the judge's output to reduce bias
- By rounding to the nearest 0.25 increment
A: By using the underlying token probabilities of the judge's output to reduce bias

Q: A vague G-Eval criteria like "is this a good answer?" is acceptable because G-Eval always produces a deterministic, reproducible score regardless of rubric specificity.
- True
- False
A: False

Q: A team needs a reproducible, fully deterministic evaluation path for a compliance-related check, not a nuanced LLM judgment call. Which DeepEval metric type is designed for this?
- G-Eval with a longer criteria string
- A DAG (deterministic decision-graph) metric
- Contextual Recall
- The hallucination metric run twice and averaged
A: A DAG (deterministic decision-graph) metric

Q: Describe the hierarchical failure-analysis approach: what do you score first, and what do you score next to pinpoint whether a RAG failure happened in retrieval or generation?
Feedback: Sample/expected answer: First score the whole trace/end-to-end output (e.g., with Answer Relevancy or a custom G-Eval metric) to confirm a real quality problem exists. Then score the individual spans separately: Contextual Precision/Recall on the retrieval span to check whether the right context was fetched, and Faithfulness on the generation span to check whether the model used the given context honestly — localizing whether the failure originated in retrieval or generation.