Title: Lecture 12 — The Fourth Pillar: Evaluating What Logs and Metrics Can't See
Course: AIINFRA 303: Capstone Project > Section: Week 12 — Observability, Evaluation, and Cost Instrumentation

📖 Lecture — The Fourth Pillar: Evaluating What Logs and Metrics Can't See

By now you've built services, containerized them, deployed them to the cloud, wired up CI/CD, served models for inference, fine-tuned them, wrapped them in agents with MCP tools, grounded them with RAG, hardened them, and started tracking their cost. This week closes the loop: how do you actually know your capstone is working, in production, over time? Traditional observability rests on three pillars: logs (discrete, timestamped events — "request received," "tool call failed"), metrics (aggregated numbers over time — request rate, p99 latency, error percentage), and traces (the end-to-end path of a single request across services, showing where time and errors accumulate). If you came up through general backend or DevOps work, these three pillars are probably second nature by now, and they remain necessary for any LLM-backed system. They tell you the service is up, fast, and not throwing exceptions. Here's the problem: none of the three pillars can tell you whether the content of an LLM response was any good. A request to your capstone's /chat endpoint can return 200 OK in 400ms with zero exceptions logged, and still have fabricated a citation, answered a completely different question than the one asked, called the wrong tool, leaked a system prompt, or quietly cost you 40x your per-request budget because the model rambled for 3,000 output tokens instead of 200. Your logs are clean. Your metrics are green. Your trace shows a fast, healthy span. And your system just failed its actual job. This is why LLM systems need a fourth pillar: automated evaluation — systematic, ongoing scoring of what the model actually produced, not just how fast it produced it. The practical move this week is to stop treating evaluation as a one-time, pre-launch activity and start treating it as a continuous production process, the same way you already treat metrics. The workflow looks like this: sample real production traffic, run automated evaluations against that sample (using an LLM-as-judge, a rubric, or a quantitative metric library), and when an example fails, save it as a permanent regression test case. Over time you accumulate a growing eval set built entirely from real failures your system actually produced — not synthetic examples you imagined in advance. Pair this evaluation loop with cost instrumentation on the same dashboard, so that "this prompt version is more accurate" and "this prompt version costs 3x more per request" are visible side by side, not discovered separately three weeks later. Two tools make this concrete and free to run locally. Langfuse is open source and self-hostable; it gives you LLM tracing, prompt versioning, and evaluation (LLM-as-judge or custom metrics) in one place, without locking you into a single model vendor's proprietary observability console. OpenTelemetry (OTEL) is the vendor-neutral standard your traces should already speak, and it now has GenAI-specific semantic conventions — standardized span attributes like gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — so that token counts and cost data ride along on the same trace as latency, consumable by whatever backend you point it at (Langfuse, Grafana Tempo, Jaeger, or a vendor tool later). For the RAG layer specifically, "the answer looked reasonable to me" is not a metric. RAGAS gives you four quantitative axes: faithfulness (is the answer actually supported by the retrieved context, or did the model hallucinate beyond it?), answer relevancy (does the answer address the question asked?), context precision (of the chunks retrieved, how many were actually useful?), and context recall (did retrieval surface the chunks that were actually needed?). Run these on a labeled eval set and you get a baseline you can defend and re-run after every retrieval or prompt change.

Pillar | Answers the question | Example tool
Logs | What happened, in order? | Structured app logs
Metrics | How much, how fast, how often? | Prometheus
Traces | Where did time/errors go across the call graph? | OpenTelemetry
Evaluation (4th pillar) | Was the output actually correct, safe, and on-budget? | Langfuse, RAGAS

Three misconceptions worth correcting directly. First: a 200 OK does not mean the response was correct. A request can be fast and error-free while giving a wrong answer, invoking the wrong tool, blowing past your cost target, or leaking context it shouldn't have surfaced — you need LLM-specific evaluation layered on top of your latency/error dashboards, not instead of them. Second: don't wait for a "perfect" observability strategy before instrumenting anything. The right sequence is to turn on auto-instrumentation now — OTEL and Prometheus exporters are largely drop-in — and layer evaluation logic in incrementally over the following weeks. Waiting for the ideal design means shipping with no visibility at all. Third: your monthly provider invoice is not cost instrumentation. A single aggregate number tells you nothing about which prompt, which feature, which user, or which model call is actually driving spend. You need per-request token and compute accounting, joined to the same trace IDs as your latency data, so a cost spike can be traced back to a specific code path the same way an error spike can. By the end of this week, your capstone should have a dashboard that shows latency, throughput, errors, per-request cost, and a running RAG quality score — all in one place, all sourced from tools you can run entirely on your own machine.