Title: Quiz 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency

Q: A concrete method for detecting embedding/query drift is to:
- Compare total request counts week over week
- Measure the cosine distance between the centroid of a baseline query set and the centroid of the current query set
- Check whether average response length has changed
- Count the number of retrieved documents per query
A: Measure the cosine distance between the centroid of a baseline query set and the centroid of the current query set

Q: If a RAG service is returning HTTP 200 responses with normal latency and typical response length, you can safely assume answer quality is fine.
- True
- False
A: False

Q: Why is tracking only average latency a poor practice for production LLM systems?
- Averages are harder to compute than percentiles
- Averages hide the tail latency (p95/p99) that a meaningful share of users actually experience
- Averages require more storage than percentile metrics
- Averages can't be broken down by route or region
A: Averages hide the tail latency (p95/p99) that a meaningful share of users actually experience

Q: Name two token-level or latency metrics (beyond a simple total-latency average) that should be instrumented per request to properly monitor cost and performance.
Feedback: Sample/expected answer: Acceptable answers include any two of: prompt tokens, completion tokens, per-request cost (by model/user/feature), TTFT (time-to-first-token), TPOT (time-per-output-token), p50/p95/p99 latency, cache-hit ratio.

Q: In the feedback-loop pattern described this week, how should human feedback (ratings/thumbs) be captured to be useful for future evaluation?
- Aggregated into a single overall satisfaction score per week
- Linked to the specific trace that produced the output it's rating
- Stored separately from any trace or request data for privacy
- Only collected from internal QA testers, not real users
A: Linked to the specific trace that produced the output it's rating

Q: Once you have dashboards and alerts set up for your production LLM system, your monitoring work is essentially complete.
- True
- False
A: False

Q: In the 2026 production monitoring architecture pattern described this week, what is the role of Evidently AI, distinct from Langfuse?
- Evidently handles prompt experiments while Langfuse computes statistical drift metrics
- Evidently computes statistical drift metrics and output-quality descriptors, while Langfuse handles trace collection and eval pipelines
- Evidently routes alerts to PagerDuty while Langfuse aggregates cost into Grafana
- They are interchangeable and either can perform all monitoring functions
A: Evidently computes statistical drift metrics and output-quality descriptors, while Langfuse handles trace collection and eval pipelines

Q: What is an Evidently "descriptor," and give one example of what it can measure on a piece of generated text.
Feedback: Sample/expected answer: A descriptor is a per-text computed metric used for output-quality monitoring; examples include sentiment, toxicity, length, or semantic similarity to a reference answer.