Title: Discussion 15 — The Dashboards Are Green, So Why Is the Chatbot Getting Worse?
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency

💬 Discussion 15 — The Dashboards Are Green, So Why Is the Chatbot Getting Worse? (20 pts)

All of your monitoring dashboards are green: uptime is 99.9%, average latency is normal, response lengths look typical. Then a support ticket comes in — a user says the chatbot has been giving worse answers for the last two weeks. Nothing in your infrastructure monitoring caught it. In a post of at least 200 words, answer:

- What specific signal (from this week's material — drift detection, cost/latency percentiles, or output-quality descriptors) would most plausibly have caught this degradation before the support ticket did, and why?
- Walk through concretely how you'd instrument for that signal: what would you measure, what would the baseline be, and what threshold or pattern would trigger an alert?
- Once the alert fires, what's the very next action it should trigger (per this week's "close the loop" idea) — and who or what should be responsible for taking it?

Then respond to at least two classmates: push on whether their proposed signal would actually have fired two weeks in, or whether it would have caught the problem just as late as the support ticket did.