🛠️ Lab 6 — Self-Hosted Tracing with Langfuse, LiteLLM, and Local vs. Cloud Models (50 pts)

Goal: Self-host Langfuse locally, instrument an agent so every trace captures LLM calls, tool executions, timing, token counts, and computed cost, then use that trace to debug a slow/expensive run. Extend the setup to confirm your traces are backend-portable via OpenTelemetry, and finish by comparing a cloud-hosted model against a fully local model side by side — using only free, self-hostable tools. Steps:

  1. Stand up Langfuse locally. Install Docker Desktop (or Docker Engine) if you don't already have it. Clone the Langfuse repository and run docker compose up from its root — this takes about 5 minutes and brings up Langfuse's web UI, database, and OTLP ingestion endpoint on your machine. Confirm you can log in to the local Langfuse UI at http://localhost:3000 and create a project.
  2. Instrument a simple agent. Using an agent you already built in an earlier week (or a small new one with at least one tool call, e.g., a calculator or web-search stub), add OpenTelemetry instrumentation so every LLM call and tool call becomes a span. Configure your OTel exporter to send traces to your local Langfuse instance's /api/public/otel endpoint using the project API keys from Step 1.
  3. Run the agent and confirm full trace capture. Send the agent 3–5 test prompts, including at least one that requires two or more tool calls. In the Langfuse UI, confirm each run appears as a trace containing nested spans for every LLM call and tool call, with input/output token counts, per-span latency, and a computed cost per call visible.
  4. Debug a slow/expensive run using only the trace. Deliberately construct one prompt that causes a slow or token-heavy run (e.g., a prompt that triggers a retry loop, a long tool response, or an oversized context). Without looking at your code, use only the Langfuse trace tree to identify which specific span is responsible for the added latency or cost. Write down what you found and how the trace told you.
  5. Route through LiteLLM to confirm backend portability. Install LiteLLM (free, open source) as a local proxy in front of your model calls. Point your agent's API calls at the LiteLLM proxy instead of calling the provider directly, and configure LiteLLM (or your OTel exporter) to continue sending OTLP traces to the same local Langfuse instance. Confirm the traces still appear correctly — this demonstrates that the exact same instrumentation could be redirected to any OTel-compatible backend without changing your agent's code.
  6. Compare a cloud model against a local model side by side. Configure two model routes through LiteLLM: one pointing to a free-tier Groq-hosted model, and one pointing to a locally running Ollama model (e.g., llama3.1:8b or similar, run via ollama serve on your own machine). Run the identical set of 3–5 test prompts from Step 3 against both models and capture the resulting traces in Langfuse for each.
  7. Build a comparison table. Using the Langfuse traces from Step 6, record input tokens, output tokens, total latency, and computed cost (where applicable — note that a local Ollama model has no per-token dollar cost, but still has compute time/latency) for the Groq model vs. the Ollama model, per prompt.

Deliverables: A short written report (1–2 pages or equivalent) containing: (1) screenshots of at least two full Langfuse trace trees — the multi-tool-call trace and the slow/expensive trace, with your written explanation of what the slow/expensive trace revealed and how you found it using only the trace; (2) confirmation that traces reached Langfuse both directly and via the LiteLLM/OTel path; (3) your Groq-vs-Ollama comparison table from Step 7 with a 3–4 sentence takeaway on the cost/latency/quality tradeoffs you observed. Submit the report plus your instrumented agent code (upload or a text-entry link to your repo).