Title: Discussion 12 — The Dashboard That Lied
Course: AIINFRA 303: Capstone Project > Section: Week 12 — Observability, Evaluation, and Cost Instrumentation

💬 Discussion 12 — The Dashboard That Lied (20 pts)

Describe a specific request to your capstone (or a hypothetical one, if you haven't hit this yet) that would pass every traditional metric — 200 status, sub-second latency, zero exceptions logged — and still represent a real failure that your system should catch. Be concrete: what did the model actually do wrong (wrong tool call, fabricated fact, off-topic answer, budget blowout, leaked context, something else), and what specific signal — a RAGAS score, an LLM-as-judge rubric, a token-cost threshold, a saved regression test — would have caught it? Then respond to at least two classmates: does their proposed detection method actually work for their stated failure, or would it miss it too?