Title: Lecture 8 — Benchmarking Serving Engines the Right Way
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 08 — Serving Engine Benchmarking Lab

📖 Lecture — Benchmarking Serving Engines the Right Way

You've now deployed inference servers and scaled them across GPUs. The natural next question every team asks is: "Which serving engine should we actually run in production?" The honest answer is always "it depends" — and this week is about learning to make that answer rigorous instead of a shrug.

The metrics that actually matter

Serving benchmarks live or die on a small, standard set of metrics. Get comfortable with all five before you touch a benchmarking tool:

Metric | What it measures | Why it matters
Time to First Token (TTFT) | Time from request submission to the first generated token arriving, including queueing + prefill + network overhead | Dominates perceived responsiveness, especially for chat UIs
Time Per Output Token (TPOT) | Average time between subsequent tokens once generation has started | Drives the "typing speed" a user feels during streaming
End-to-end latency | Total wall-clock time from request to final token | What a synchronous caller actually waits for
Throughput (tokens/sec) | Tokens generated per second across the system, excluding TTFT | Capacity metric — how much work the engine can push
Requests per second (RPS) | Completed requests per second under load | Capacity metric at the request level, useful for capacity planning

Notice that TTFT and TPOT answer a user experience question, while throughput and RPS answer a capacity question. A serving engine can look fantastic on one axis and mediocre on the other — which is exactly why benchmarking requires more than one number.

The throughput/latency tradeoff is not optional

Here's the core mental model for this week: pushing throughput up (larger batches, more concurrent requests) almost always pushes per-request latency up too. Bigger batches mean better GPU utilization and more tokens/sec system-wide, but each individual request now waits longer in the queue and competes for compute during decoding. This is not an engine bug — it's a fundamental resource-contention tradeoff. The practical implication: the "best" serving engine depends on your SLO, not on a leaderboard. A team building a low-traffic internal tool with generous latency tolerance should optimize differently than a team building a customer-facing chat product with a strict TTFT SLO. Benchmarking's job is to characterize the curve — throughput at various concurrency levels, and the latency cost paid at each point on that curve — not to find one magic number.

Tooling: don't reinvent the harness

Two tools you'll use in this week's lab:

- NVIDIA GenAI-Perf is a purpose-built LLM inference benchmarking client. It measures tokens/sec, latency percentiles, and TTFT against any OpenAI-compatible endpoint — meaning it works identically against vLLM, TGI, Triton, or NIM. This makes it a strong cross-engine harness because the measurement methodology stays constant while the backend changes.
- vLLM's built-in bench CLI (vllm bench latency, vllm bench serve, vllm bench throughput) covers three distinct scenarios: single-batch latency, online serving throughput against a live server, and offline batch throughput. It reports both request/token throughput and latency percentiles natively.

Correcting three common misconceptions

1. "I'll just report average latency." Don't. Averages hide pain. A model with a low mean latency can still have a brutal P99 that makes 1% of your users wait 10x longer. Always report P50, P90, and P99 — tail latency is what users actually feel, and it's often what breaks your SLO first.
2. "TTFT and TPOT can be folded into one per-token average." They can't be compared naively across tools. Different benchmarking tools disagree on whether the first token's latency should be included in the per-token average or reported separately. Before comparing numbers across two tools (or two papers), check each tool's methodology — otherwise you're comparing apples to a different tool's oranges.
3. "My short test prompts give me a representative tokens/sec number." They don't. Short, easy prompts reduce prefill work and TTFT, which inflates your throughput number relative to real production traffic — where prompts are long, variable, and expensive to prefill. Always benchmark with realistic prompt-length distributions (this week's lab uses the ShareGPT dataset for exactly this reason).

What the published numbers actually show

Representative cross-engine benchmarks report TensorRT-LLM outperforming vLLM at every concurrency level tested — roughly 8% faster at 1 concurrent request, climbing to about 13% faster at 50 concurrent requests. Meanwhile, TGI v3 dramatically outperforms vLLM specifically on very long prompts (200k+ tokens), driven by its prefix caching design. Notice the pattern: each engine's advantage is conditional — on concurrency level, on prompt length, on caching behavior. That's the whole point of this week. A single "X is the fastest engine" headline is almost always missing the workload context that made it true. Head into the Midterm and this week's lab treating every benchmark claim — including the ones above — as a hypothesis to verify with your own methodology, not a fact to memorize.