Title: Quiz 14 — Agent Evaluation and Observability
Course: AIINFRA 300: Agentic AI & the Model Context Protocol (MCP) > Section: Week 14 — Agent Evaluation and Observability

Q: An outcome-level metric alone is sufficient to evaluate an agent, since all that ultimately matters is whether the final answer is correct.
- True
- False
A: False

Q: Which of the following is NOT one of the trajectory-level dimensions discussed this week?
- Step efficiency
- Tool correctness
- Model parameter count
- Plan adherence
A: Model parameter count

Q: What distinguishes "Agent-as-a-Judge" from a standard single-call LLM-as-judge setup?
Feedback: Sample/expected answer: Agent-as-a-Judge uses a full agentic system (with its own tools/reasoning) to critique another agent's entire trajectory, including decision points and intermediate actions — not just a single-shot judgment on the final output.

Q: MCP-Bench scores agents across which three dimensions?
- Latency, cost, and token count
- Task completion quality, tool-selection rationale, and planning effectiveness
- Code coverage, test pass rate, and lines of code
- User satisfaction, retention, and NPS
A: Task completion quality, tool-selection rationale, and planning effectiveness

Q: DeepEval requires a labeled ground-truth answer for every test case before it can score tool selection or answer relevance.
- True
- False
A: False

Q: Name two practices for designing a judge prompt that produces more reliable scores.
Feedback: Sample/expected answer: Any two of: an explicit rubric, few-shot examples, structured JSON output, requiring cited evidence before a numeric score.

Q: In expert domains such as law, medicine, or mental health, LLM-judge agreement with human experts has been found to:
- Exceed 95%, higher than inter-expert agreement
- Drop to roughly 64–68%, below inter-expert agreement baselines
- Be exactly equal to inter-expert agreement in all domains
- Be irrelevant since judges never disagree with experts
A: Drop to roughly 64–68%, below inter-expert agreement baselines

Q: Passing every JSON-schema/structural check on an agent's output is equivalent to verifying that the output is semantically correct and truthful.
- True
- False
A: False