Title: Lab 12 — Dual Tracing: Instrumenting a RAG App with Phoenix and Langfuse
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 12 — Tracing and Observability: Langfuse and Arize Phoenix

🛠️ Lab 12 — Dual Tracing: Instrumenting a RAG App with Phoenix and Langfuse (50 pts)

Goal: Instrument the same small RAG application with two free, self-hosted observability tools — Arize Phoenix and Langfuse — collect traces, run automated evaluations over those traces, and compare the developer experience of each. Steps:

1. Prepare a small RAG app. Reuse (or build a minimal version of) a RAG pipeline from AIINFRA 300: a retriever over a small local document set plus an LLM call to generate an answer. Keep it simple — 3–5 sample questions is enough.
2. Stand up Phoenix locally. In a Jupyter notebook or local Python environment, run pip install arize-phoenix openinference-instrumentation-openai (swap in the OpenInference package matching your LLM client). Call register() and .instrument() on your client library. Confirm the Phoenix UI is reachable at localhost:6006.
3. Run your RAG app through Phoenix. Execute your sample questions so spans are captured automatically via OpenInference/OpenTelemetry. In the Phoenix UI, inspect the resulting trace tree and confirm you can see retrieval and generation as distinct nested spans.
4. Pull traces programmatically and run evals. Use get_spans_dataframe() to pull your captured spans into a DataFrame. Run Phoenix's built-in Relevance, Q&A Correctness, and Hallucination evals over the traces. Save the resulting eval scores (screenshot or exported CSV).
5. Stand up Langfuse locally. Clone the Langfuse repo, set NEXTAUTH_SECRET, SALT, and ENCRYPTION_KEY, and run docker-compose up -d. Create your admin account on first login.
6. Re-instrument the same RAG app with Langfuse. Add from langfuse import observe, get_client and decorate your retrieval and generation functions with @observe. Re-run your same sample questions.
7. Inspect the Langfuse trace view. Confirm your traces show the expected nested Span/Generation structure, and that token counts, latency, and (if available) cost appear on the Generation node.
8. Write your comparison. In 300–500 words, compare the two tools on: (a) setup friction, (b) instrumentation code required, (c) quality/readability of the trace visualization, and (d) how easy it was to get evaluation scores out of each. State which you'd choose for a small team's first production observability setup, and why.

Deliverables: A short writeup (300–500 words) covering your Phoenix eval results and your Phoenix-vs-Langfuse comparison, plus supporting screenshots or exported files (trace views, eval score tables) and any code/config you used, submitted as a file upload or pasted as text.