Title: Lecture 15 — Watching a RAG System That Never Stops Talking
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency

📖 Lecture — Watching a RAG System That Never Stops Talking

Here's an uncomfortable truth: a RAG system can be having a bad week while every infrastructure dashboard says everything is fine. Misconception check: if the service returns HTTP 200s with normal latency and reasonable response length, quality must be fine. It isn't. Infra metrics measure whether the plumbing works — did the request complete, how long did it take. None of that tells you whether retrieved chunks were relevant or the answer was correct. Relevance can quietly decay for weeks while ops dashboards stay green, because nothing is watching the actual content of what goes in and out. That gap is exactly what this week's tools close. Embedding and query drift. Your vector index was built on a snapshot of documents and tuned against a snapshot of expected queries. Production doesn't hold still. Users start asking about a new product line, a slang term takes over, or a seasonal topic shifts vocabulary — and the distribution of incoming queries moves away from what the index was built to serve well. This is embedding/query drift. A concrete, teachable detection method: compute the centroid (average embedding vector) of a baseline period's queries, then compute the centroid of the current period's queries, and measure the cosine distance between them. When that distance crosses a set threshold, you have a quantitative, alertable signal that the input distribution has moved — before relevance visibly collapses. Cost and latency, at the right granularity. The second misconception to retire: tracking one average latency number and one total cost number. Averages hide the tail — and the tail is what users actually feel. Track p50, p95, and p99 latency by route and by region, plus time-to-first-token (TTFT) and time-per-output-token (TPOT) for streaming responses. On the cost side, instrument at the span level: log prompt tokens, completion tokens, and per-request cost, attributed to the model, the user, and the feature that triggered the call. That granularity is what lets you answer "why did the bill triple" or "why do 5% of requests feel slow" instead of just observing that it happened.

Metric category | What to log | Why averages fail here
Latency | p50 / p95 / p99, TTFT, TPOT, by route/region | Averages hide the slow tail users notice most
Cost | Prompt vs. completion tokens, per-request \$ by model/user/feature, cache-hit ratio | A single total obscures which feature or prompt pattern drives spend
Drift | Query/embedding centroid cosine distance vs. baseline | Silent input shift degrades retrieval while infra metrics stay green
Quality | Evidently descriptors: sentiment, toxicity, length, semantic similarity to reference | Response-time and status codes say nothing about answer quality

Closing the loop with feedback. The third misconception: monitoring is "done" once dashboards and alerts exist. Dashboards only help if they connect to action. Predefine what each threshold breach actually triggers — a prompt revision, a retrieval retune, a model switch — so an alert firing isn't a dead end. Then go one step further and close the loop with real user feedback: capture thumbs-up/down or ratings and link that feedback directly to the trace that produced the output (tools like MLflow expose this via mlflow.log_feedback()), and let eval failures auto-create a team task rather than sitting unread in a log. Over time, that captured feedback becomes your evaluation dataset — built from real usage instead of a hand-written test set that goes stale. The maturity ladder runs: structured tracing → dashboards and alerts → automated evals → feedback loops that generate new eval data. Most teams stop at dashboards. The teams that actually keep quality up push to the last two rungs. The 2026 pattern in practice composes tools rather than looking for one that does everything: Langfuse handles trace collection, eval pipelines, and prompt experiments; Evidently AI computes statistical drift metrics and per-text quality "descriptors" (sentiment, toxicity, length, semantic similarity to a reference); alerts route by severity — P1 to PagerDuty, P2/P3 to Slack; and cost is tracked via custom span attributes rolled up into Grafana dashboards. No single tool does all of this well — composition is the point, not a compromise.