Title: Discussion 14 — When the Judge and the Human Disagree
Course: AIINFRA 300: Agentic AI & the Model Context Protocol (MCP) > Section: Week 14 — Agent Evaluation and Observability

💬 Discussion 14 — When the Judge and the Human Disagree (20 pts)

Prompt: Suppose your LLM judge scores an agent trajectory as a success, but a human reviewer looking at the same transcript calls it a failure (or vice versa). Whose verdict should count — and more importantly, how would you design your evaluation system to resolve that kind of disagreement systematically, rather than deciding it case by case every time it comes up? Ground your answer in something concrete from this week: the reliability numbers for LLM judges in expert domains, the layered-evaluation idea (offline benchmarks plus online production-trace scoring), or your own Lab 14 experience comparing your judgment to DeepEval's scores. Propose at least one specific mechanism (e.g., a disagreement threshold that routes to human review, a periodic recalibration process, a confidence-weighted scoring scheme) rather than just saying "use both." Post your initial response, then respond to at least two classmates — ideally engaging with a proposed mechanism different from your own and pressure-testing whether it would actually hold up in a high-stakes domain like law or medicine.