You've now deployed inference servers and scaled them across GPUs. The natural next question every team asks is: "Which serving engine should we actually run in production?" The honest answer is always "it depends" — and this week is about learning to make that answer rigorous instead of a shrug.
Serving benchmarks live or die on a small, standard set of metrics. Get comfortable with all five before you touch a benchmarking tool:
| Metric | What it measures | Why it matters |
|---|---|---|
| Time to First Token (TTFT) | Time from request submission to the first generated token arriving, including queueing + prefill + network overhead | Dominates perceived responsiveness, especially for chat UIs |
| Time Per Output Token (TPOT) | Average time between subsequent tokens once generation has started | Drives the "typing speed" a user feels during streaming |
| End-to-end latency | Total wall-clock time from request to final token | What a synchronous caller actually waits for |
| Throughput (tokens/sec) | Tokens generated per second across the system, excluding TTFT | Capacity metric — how much work the engine can push |
| Requests per second (RPS) | Completed requests per second under load | Capacity metric at the request level, useful for capacity planning |
Notice that TTFT and TPOT answer a user experience question, while throughput and RPS answer a capacity question. A serving engine can look fantastic on one axis and mediocre on the other — which is exactly why benchmarking requires more than one number.
Here's the core mental model for this week: pushing throughput up (larger batches, more concurrent requests) almost always pushes per-request latency up too. Bigger batches mean better GPU utilization and more tokens/sec system-wide, but each individual request now waits longer in the queue and competes for compute during decoding. This is not an engine bug — it's a fundamental resource-contention tradeoff. The practical implication: the "best" serving engine depends on your SLO, not on a leaderboard. A team building a low-traffic internal tool with generous latency tolerance should optimize differently than a team building a customer-facing chat product with a strict TTFT SLO. Benchmarking's job is to characterize the curve — throughput at various concurrency levels, and the latency cost paid at each point on that curve — not to find one magic number.
Two tools you'll use in this week's lab:
bench CLI (vllm bench latency, vllm bench serve, vllm bench throughput) covers three distinct scenarios: single-batch latency, online serving throughput against a live server, and offline batch throughput. It reports both request/token throughput and latency percentiles natively.Representative cross-engine benchmarks report TensorRT-LLM outperforming vLLM at every concurrency level tested — roughly 8% faster at 1 concurrent request, climbing to about 13% faster at 50 concurrent requests. Meanwhile, TGI v3 dramatically outperforms vLLM specifically on very long prompts (200k+ tokens), driven by its prefix caching design. Notice the pattern: each engine's advantage is conditional — on concurrency level, on prompt length, on caching behavior. That's the whole point of this week. A single "X is the fastest engine" headline is almost always missing the workload context that made it true. Head into the Midterm and this week's lab treating every benchmark claim — including the ones above — as a hypothesis to verify with your own methodology, not a fact to memorize.