Title: Lecture 15 — Seeing What Your GPUs Are Actually Doing (and What It Costs You)
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 15 — Observability, Cost Analysis, and Right-Sizing

📖 Lecture — Seeing What Your GPUs Are Actually Doing (and What It Costs You)

Up to now in this course you've focused on getting models served correctly and reliably. This week we turn to a different question: is your GPU fleet actually being used well, and how much money is that costing or saving you? These two questions — observability and cost — are really the same question asked twice, because you can't manage a cost you can't see, and GPUs are the single most expensive line item in any inference deployment. The observability maturity path Most ML teams don't wake up one day with a full Prometheus/Grafana stack. Observability grows in stages that match the scale of the problem you're solving:

Stage | Tool | When you use it | What it tells you
1. Development | nvidia-smi | Quick, single-GPU sanity checks while iterating locally | Instantaneous utilization, memory used, temperature, running processes
2. Deep debugging | PyTorch Profiler | You suspect a specific kernel, op, or data-loading step is slow | Per-op timing, CUDA kernel traces, memory allocation timelines
3. Production fleet | DCGM + Prometheus + Grafana | You're running many GPUs/nodes and need historical, alertable, dashboardable telemetry | Time-series utilization, memory bandwidth, ECC errors, temperature, power draw — across the whole fleet

This progression matters pedagogically: nvidia-smi is the right tool for a single developer checking one box, but it does not scale to a fleet, has no history, and can't page anyone at 2 a.m. NVIDIA's DCGM (Data Center GPU Manager) exporter closes that gap. It runs as a small agent per node, collects low-level GPU telemetry (utilization, memory bandwidth usage, kernel execution time, ECC errors, temperature, power), and exposes it all at a standard Prometheus /metrics HTTP endpoint. Prometheus scrapes that endpoint on an interval and stores it as time series; Grafana reads from Prometheus to render dashboards and power alerts. This is the same pattern you already know from application observability — it's just pointed at the GPU driver instead of your app. On top of hardware telemetry, your serving engine adds its own metrics. vLLM, for example, exposes vllm:gpu_cache_usage_perc (how full the KV cache is) and vllm:num_requests_waiting (how many requests are queued because the engine can't schedule them yet). These serving-level metrics are often more actionable than raw GPU utilization because they tell you why the GPU is busy or idle — a full KV cache with a growing wait queue points at a capacity problem; a near-idle GPU with an empty queue points at an under-provisioned traffic pattern, not a broken deployment. Misconception 1 corrected: 100% GPU utilization does not mean the GPU is being used well. This is the single most important idea this week, so let's be precise about it. The "GPU-Util" number reported by nvidia-smi and DCGM measures only the percentage of time during the sampling window that at least one kernel was executing on the GPU. It is a busy/idle duty-cycle measurement, not a measurement of computational efficiency. A GPU can show 100% utilization while running a poorly parallelized kernel that uses 5% of available compute throughput — the GPU is "busy" the entire time, just not doing much useful work per cycle. This is why utilization dashboards must always be read alongside throughput metrics (tokens/sec, requests/sec) and, ideally, achieved FLOPs versus theoretical peak FLOPs. Utilization tells you the GPU isn't sitting idle; it does not tell you the work is efficient. Misconception 2 corrected: most organizations do not run GPUs anywhere near capacity. New teams often assume that because GPUs are scarce and expensive, everyone is squeezing every drop out of them. Industry surveys tell a very different story: over 75% of organizations run their GPUs below 70% utilization even at peak load, and only about 7% ever break above 85% utilization. Idle and semi-idle GPU time is the norm, not the exception — which is exactly why right-sizing and utilization improvement is such high-leverage work. Consider a 100-GPU cluster billed at \$2/hr per GPU: improving fleet utilization by just 10 percentage points (by consolidating workloads, improving batching, or turning off idle capacity) saves roughly \$175,000 per year. At fleet scale, utilization percentage points are literally worth six figures. Misconception 3 corrected: "memory utilization" is not one number. Beginners tend to watch a single memory percentage and assume it captures the whole picture. In reality GPU memory has two independent dimensions you must track separately:

Dimension | What it measures | Symptom when it's the bottleneck
Memory capacity (VRAM allocated) | How much of the GPU's total memory is currently occupied by model weights, KV cache, and activations | Out-of-memory errors, inability to increase batch size or context length
Memory bandwidth | How fast data moves between GPU memory and compute units (measured in GB/s, often shown as "memory copy utilization" in DCGM) | High GPU-Util with low actual throughput — the compute units are stalled waiting on data

A deployment can have plenty of free VRAM (capacity) while still being bandwidth-starved, because bandwidth is consumed continuously by every kernel launch, not just by how much data is resident. Conflating these two dimensions means teams misdiagnose "the GPU has memory to spare" as "the GPU has room to grow," when the real constraint is how fast data can be shuttled to the compute cores. Cost optimization: precision is the biggest lever you have Once you can see utilization and bandwidth clearly, the next question is cost. The single biggest cost optimization available to most teams is reducing model weight precision. A Llama-3-70B model served in BF16 requires a minimum of two H100 80GB GPUs (roughly \$2.69/hr each on-demand) just to fit the weights and KV cache. The same model quantized to a 4-bit format such as AWQ shrinks enough to run on dual RTX A6000 GPUs at roughly \$0.49/hr each — a dramatically cheaper class of hardware. Quantization, batching, and right-sizing GPU count are not separate tricks; they compound. This week's lab asks you to put real numbers behind that compounding effect. The throughline for this week: you cannot right-size what you cannot measure, and you cannot trust a measurement you do not understand. Build the observability stack first, read it correctly (utilization ≠ efficiency, capacity ≠ bandwidth), and only then make cost decisions.