Title: Lecture 13 — Choosing (and Combining) Your Observability Stack
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 13 — Observability at Scale: LangSmith, OpenLLMetry, and MLflow

📖 Lecture — Choosing (and Combining) Your Observability Stack

Last week you stood up self-hosted Langfuse to trace a RAG pipeline. This week we zoom out to the broader observability landscape you'll actually choose from at production scale, because Langfuse is only one of several serious options, and the right answer is rarely "pick one and never look at the others again." Three approaches, three philosophies. OpenLLMetry, built by Traceloop, is an open-source instrumentation-only library. It doesn't ship a UI, an evaluation product, or a prompt playground — it just wraps your LLM, chain, agent, tool, and retriever calls in standard OpenTelemetry (OTel) spans and exports them wherever you point it. That makes it maximally flexible: you own the trace store, you choose the backend, and you're never locked into anyone's pricing model. The tradeoff is that you have to bring your own UI and evaluation tooling, typically by pointing OpenLLMetry at Langfuse, Grafana, Honeycomb, or another OTel-compatible backend. LangSmith, built by LangChain, sits at the opposite end. It's a closed, proprietary platform with the deepest native integration into LangChain and LangGraph of any tool on the market, plus a genuinely polished UI for trace inspection, dataset curation, and prompt evaluation. The catch at scale: self-hosting LangSmith requires an Enterprise contract and your own Kubernetes cluster. If you want the hosted convenience, you pay per trace as volume grows. MLflow occupies useful middle ground. It's open-source and self-hostable like OpenLLMetry, but it ships a full observability and experiment-tracking UI out of the box, and its LLM/GenAI tracing support now covers 60+ frameworks with one-line autologging. With 30M+ monthly downloads, it's also the most battle-tested of the three in general ML contexts, and it lets you bring your own Postgres, MySQL, or RDS instance for storage rather than committing to a vendor's database. The concrete pricing reality. LangSmith's 2026 tiers make the scale tradeoff explicit:

Tier | Price | Base traces included | Overage | Notes
Developer | Free | 5,000 traces/month | Not available | Good for coursework and prototypes
Plus | \$39/seat/month | 10,000 traces included | Priced per additional 1,000 traces | Per-seat cost stacks with team size
Enterprise | Custom | Unlimited seats | Custom | Required for self-hosting

Across all tiers, base traces are retained for 14 days; extended (longer-retention) traces are kept for 400 days at additional cost. Notice what this means operationally: trace volume is a metered, recurring expense. A team whose agent traffic grows 10x doesn't just need a bigger server — it needs a bigger LangSmith bill, or an Enterprise contract plus the Kubernetes operations burden of self-hosting. MLflow and OpenLLMetry sidestep that by letting you scale storage the way you already scale any other database: horizontally, on infrastructure you control. Two lines of code, either way. Both open options are strikingly easy to adopt:

# OpenLLMetry
pip install traceloop-sdk
# then, once at startup:
from traceloop.sdk import Traceloop
Traceloop.init()

# MLflow
import mlflow
mlflow.langchain.autolog()

Both calls instrument chains, LLM calls, agents, tools, and retrievers automatically — no manual span-wrapping required. That's the same automatic-capture promise LangSmith and Langfuse make, just without a proprietary SDK lock-in. Correcting two common misconceptions. First: "At production scale I should capture full verbose traces for every request, the same way I do in dev." This does not hold up once you're running agents in production. Agent traces are deeply nested — a single multi-step tool-calling conversation can carry megabytes of payload once you include full retrieved documents, intermediate reasoning, and tool outputs. The correct pattern is to scope capture deliberately: broad, verbose tracing in dev where volume is low and you need maximum debuggability, then sampling (e.g., trace 5–10% of requests) and tagging (e.g., only trace requests flagged as errors or from a canary cohort) in staging and production. Strip or mask large and sensitive fields — full retrieved document text, PII, raw embeddings — before export, not after. Second: "These tools are competitors, so I have to pick exactly one." In practice they interoperate through OpenTelemetry. OpenLLMetry can export to LangSmith. LangSmith can both ingest and emit OTel spans. MLflow autologs 40+ libraries and can sit alongside either. Many teams layer tools deliberately: one tool for broad operational telemetry (uptime, latency, error rate) and a different, more RAG-specialized tool for retrieval quality and answer evaluation. The question isn't "which one tool wins" — it's "which layer does each tool serve, and where do the OTel seams connect them." The practical takeaway for your own stack decisions: start by asking how much you value UI polish and native LangChain integration (favors LangSmith) versus long-term self-hosting rights and vendor-neutral storage (favors OpenLLMetry/MLflow), then design your capture strategy — verbosity, sampling, payload scrubbing — as a first-class engineering decision, not an afterthought.