By now you can build an agent that calls tools, chains LLM calls, and ships to a container. But once that agent is running in production, a new problem shows up: it becomes a black box. Requests go in, responses come out, and somewhere in the middle your cloud bill climbs, your latency drifts, and — every so often — the agent quietly fails at its job while every individual system call underneath it reports success. This week is about instrumenting that black box so you can see cost, latency, and correctness clearly enough to debug it.
Early in your career you probably reached for print() or a log line whenever something misbehaved. That works for a single API call. It falls apart for an agent, because an agent's execution isn't a single call — it's a tree. One user request might trigger an LLM call, which triggers a tool call, which triggers another LLM call to interpret the tool's result, which triggers a retry, which triggers yet another tool call. Flat logs give you a pile of timestamped lines with no shared structure connecting them. You can't easily answer "how long did the whole request take," "which step cost the most tokens," or "which tool call caused the retry." The fix the industry converged on is distributed tracing, borrowed from microservices observability and adapted for LLM workloads. A single user request becomes a trace. Each unit of work inside it — an LLM call, a tool invocation, a retrieval step — becomes a span. Spans nest inside each other, forming a tree that mirrors the actual call structure of your agent. When you open a trace viewer, you're looking at exactly the tree your agent walked: which tool got called, in what order, how long each step took, and how many tokens each LLM call consumed.
The problem with building this yourself is that every agent framework (LangChain, LlamaIndex, a hand-rolled loop) and every observability vendor (Langfuse, LangSmith, Phoenix, TruLens) would otherwise invent its own span shape and attribute names. That makes cost and latency numbers impossible to compare across tools, and locks you into whichever vendor you started with. OpenTelemetry (OTel) — already the standard for distributed tracing in traditional software — has extended its semantic conventions to cover GenAI workloads specifically. The key attributes you'll see on any properly instrumented LLM span:
| Attribute | Captures | Why it matters |
|---|---|---|
gen_ai.usage.input_tokens |
Tokens sent to the model | Primary driver of prompt/context cost |
gen_ai.usage.output_tokens |
Tokens generated by the model | Primary driver of completion cost |
gen_ai.request.model |
Model name/version requested | Lets you compare cost/latency by model choice |
gen_ai.system |
Which provider/framework generated the span | Vendor-neutral filtering across a multi-framework stack |
| Span duration | Wall-clock time for that call | Root cause of latency, isolated to one step |
Because these attributes are standardized at the instrumentation layer — not inside any one vendor's dashboard — the same trace data can flow into OpenLLMetry, Phoenix, TruLens, Langfuse, or LangSmith with identical field names. You compute cost and latency once, consistently, regardless of which framework produced the call or which backend you view it in.
A single agent request often crosses framework boundaries: your orchestration code calls an LLM through one SDK, a retrieval step through another, and a custom tool through plain Python. For all of those spans to end up nested under one trace instead of scattering into disconnected fragments, you need context propagation. OpenTelemetry's Baggage mechanism carries key-value context (like a user ID, a feature name, or a trace ID) across every span boundary in the call, and a BaggageSpanProcessor copies that baggage onto each span automatically. This is what lets you later ask "show me every trace for the checkout-assistant feature" instead of guessing which spans belong together. Because Langfuse, LangSmith, and Phoenix all accept traces over the OTLP (OpenTelemetry Protocol) wire format natively, you are not locked into instrumenting for one specific backend. Point your OTLP exporter at a different endpoint and the same trace data lands somewhere else — no code change in your agent.
Langfuse (MIT-licensed, 28,000+ GitHub stars) is the platform you'll self-host in this week's lab. It ingests OTel traces directly at /api/public/otel, so any OTel-instrumented agent can send data to it with zero custom SDK glue. Beyond tracing, Langfuse bundles evals, prompt management, a testing playground, and dataset tooling — the pieces you need to go from "I can see what happened" to "I can systematically test whether my prompt changes made things better or worse."
When you open a trace in Langfuse (or any agent observability platform), you're looking at a tree of nested spans: the top-level agent run, containing LLM calls, tool calls, and retries as children. This tree is the primary debugging artifact for non-deterministic agents, because it lets you localize a problem to one specific step instead of guessing across the whole run. Did the agent make one expensive LLM call, or twenty cheap ones? Did a single tool call take 8 seconds, or did the agent call it 6 times? The trace tree answers this in seconds; a wall of flat logs does not.
Instrumenting cost and latency inside your own code is one approach. Helicone offers a complementary one: a one-line proxy swap (point your API base URL at Helicone instead of the provider directly) that adds request logging, cost tracking, and analytics without touching your call sites. Helicone prices requests using an open-source cost repository covering 300+ models, or exact vendor-negotiated costs via its Model Registry v2 when operating as a full AI Gateway. This is a good option when you want cost visibility fast, without restructuring your instrumentation.
Here is the most important idea this week, and the one most new practitioners get wrong: healthy infrastructure metrics do not mean your agent is working. A trace can show every span returning a 200-equivalent status, normal latency, and reasonable token counts — and the agent can still have completely failed the user's task. Imagine an agent stuck calling the same search tool 18 times because it never receives the signal that its query returned no useful results. Every one of those 18 tool-call spans looks perfectly healthy: correct status, expected latency, no errors. Nothing in the trace's structure tells you the agent is looping without progress. That requires semantic evaluation — did the output actually satisfy the task — which is a different layer of observability from spans, latency, and token counts. Structural observability is necessary (you cannot debug what you cannot see), but it is not sufficient.
"If spans, latency, and token counts look healthy, the agent is working." This is false, and it's the single most common mistake teams make when they first add tracing. Structural health tells you the plumbing worked — the API calls succeeded, nothing timed out, nothing threw an exception. It tells you nothing about whether the agent's output was correct or whether it made progress toward the user's goal. An agent can loop 18 steps calling the same tool with the same unproductive query, and every span in that trace will look green. Catching this requires layering semantic checks — task-completion evals, output-quality scoring, or a human review of the final answer — on top of structural tracing, not instead of it. "Total monthly token count tells you what's wrong." A single aggregate number is a reporting metric, not a debugging tool. If your token spend jumps 40% this month, that total alone doesn't tell you whether the cause is prompt bloat (your system prompt or few-shot examples grew), context pressure (conversations are running longer before hitting a context limit), or a runaway loop (an agent retrying or looping unproductively). Each of those root causes needs a different fix — trimming a prompt, summarizing/truncating context, or fixing an agent's termination logic — and each requires you to drill into per-call and per-step token breakdowns, attributed back to the feature or endpoint that originated the request, not just a monthly total.
The workflow this week teaches is: instrument at the span level using OTel GenAI conventions so your numbers are vendor-neutral → propagate context with Baggage so multi-framework calls land in one trace → ship those traces to an open-source backend like Langfuse over OTLP → read the trace tree to localize expensive or slow steps → and always pair that structural view with a semantic check on whether the agent actually succeeded. Get comfortable with this loop now; it's the same loop you'll use to debug every agent you deploy for the rest of this program and beyond.