Title: Lab 15 — Drift Reports and a Production Monitoring View
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 15 — Production Monitoring: Drift Detection, Feedback Loops, Cost, and Latency

🛠️ Lab 15 — Drift Reports and a Production Monitoring View (50 pts)

Goal: Build two connected, fully free and self-hosted monitoring artifacts: (1) an Evidently drift report that detects embedding/query drift between a reference query set and a simulated "current" (shifted) query set, and (2) a Langfuse production monitoring view over the RAG traces you've built across the course — complete with user-feedback scores and cost/latency trend tracking — so you experience the full monitoring-to-feedback loop end to end.

Steps

1. Set up your environment. Install the free, open-source evidently Python package (pip install evidently) and confirm you have a self-hosted Langfuse instance running from earlier weeks (or spin one up locally via the Langfuse Docker Compose setup). No paid tiers or hosted SaaS accounts are required for this lab.
2. Assemble your reference set. Pull (or reuse from a prior week) a "reference" set of query embeddings and the document embeddings your vector index was built on — this is your baseline distribution.
3. Simulate query drift. Create a "current" set of queries that intentionally shifts away from the reference set — for example, swap in queries about a new topic area, a different language, or noticeably different vocabulary/phrasing than your original corpus supports. Embed these with the same embedding model.
4. Run centroid distance detection. Using Evidently, compute the centroid (mean embedding vector) of the reference set and of the current set, then calculate the cosine distance between the two centroids. Set a threshold (document your reasoning for the number you pick) and flag whether this run would trigger a drift alert.
5. Run the domain-classifier method as a cross-check. Also use Evidently's embedding drift detection to train a simple classifier that tries to distinguish reference vs. current embeddings — if it can distinguish them well, that's independent evidence of drift. Generate an Evidently drift report (HTML output) showing both methods side by side.
6. Add output-quality descriptors. Apply at least two Evidently descriptors (e.g., sentiment, toxicity, length, or semantic similarity to a reference answer) to a small batch of your RAG system's generated responses, and note what each descriptor would catch that a status-code check would miss.
7. Build the Langfuse monitoring view. In your self-hosted Langfuse instance, pull in RAG traces generated across the course (or generate a fresh simulated traffic run of 20–30 requests). Confirm each trace captures prompt/completion tokens, latency, and cost.
8. Attach feedback to traces. For at least 10 traces, attach a user-feedback score (thumbs up/down or a 1–5 rating) tied to that specific trace ID — simulating what a real user-facing feedback widget would send.
9. Track cost and latency trends. In Langfuse, review (or export/chart) the token-cost and latency trend across your simulated traffic run, and identify the p95/p99 latency, not just the average.
10. Close the loop. Write a short plan (half a page) describing what concrete action each signal should trigger: what happens when the drift threshold is crossed, what happens when feedback is consistently negative on a trace pattern, and how negative-feedback traces would get converted into a new evaluation dataset.

Deliverables:

Submit (a) your Evidently drift report (HTML export or screenshot set) with the centroid cosine distance and domain-classifier results and your threshold justification, (b) screenshots of your Langfuse monitoring view showing traces with attached feedback scores and the cost/latency trend, and (c) your one-page closed-loop action plan, all as a single upload or linked document.