Title: Lab 13 — Three-Way Observability Shootout
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 13 — Observability at Scale: LangSmith, OpenLLMetry, and MLflow

🛠️ Lab 13 — Three-Way Observability Shootout (50 pts)

Goal: Instrument the same RAG application with two free/local, vendor-neutral observability tools — OpenLLMetry and MLflow — and compare their trace views side by side. Optionally add a third, hosted option (LangSmith's free Developer tier) to see how a proprietary platform's UI differs from the two open options. Steps:

1. Reuse the RAG application and self-hosted Langfuse instance from your Week 12 lab (local Docker deployment). Confirm Langfuse is still running and reachable.
2. In a new virtual environment or your existing project environment, install the Traceloop SDK: pip install traceloop-sdk.
3. At the top of your RAG app's entry point, initialize OpenLLMetry and configure it to export via OTel to your self-hosted Langfuse instance's OTLP endpoint: from traceloop.sdk import Traceloop then Traceloop.init(), setting the appropriate TRACELOOP_BASE_URL/OTLP environment variables to point at Langfuse (see the Resources section for the exact cookbook).
4. Run 5–10 varied queries through your RAG app (short questions, multi-turn conversations, at least one query that triggers a retrieval miss). Confirm the traces land in Langfuse and inspect the span structure: retriever calls, LLM calls, and any tool calls.
5. In a separate run (or a duplicated copy of the app), add MLflow autologging instead: import mlflow then mlflow.langchain.autolog(). Start the local MLflow UI (mlflow ui) and re-run the same 5–10 queries.
6. Compare the two trace views in a short written comparison (300–500 words): What span/attribute information did each tool capture automatically? Which was easier to read for a multi-step agent trace? Which would you trust more for a debugging session at 2am?
7. Optional (bonus, up to +5 pts): Sign up for the free LangSmith Developer tier (no cost, 5,000 base traces/month) and run the same queries a third time with LangSmith's LangChain integration enabled. Add a paragraph comparing its UI and ease of setup to the two open tools, and note how you'd expect the 5,000-trace/month ceiling to affect a real team.
8. Write a one-paragraph "production plan": given everything you've seen this week, describe which tool(s) you would deploy for a RAG app expecting moderate but growing traffic, and why — referencing at least one concrete pricing or self-hosting fact from the lecture.

Deliverables: Submit your instrumented code (both the OpenLLMetry and MLflow integration points), one screenshot each of the Langfuse trace view and the local MLflow trace UI (and the LangSmith UI if you did the bonus), your written comparison, and your production plan paragraph.