📖 Lecture — Percentiles, Bottlenecks, and the Right-Sizing Method

By now your capstone has a working inference endpoint — containerized, deployed, maybe fine-tuned, maybe wrapped in a RAG pipeline or an agent. It works when you hit it once. The question this week answers is: what happens when fifty people hit it at once, and how do you know if that's fine or a problem? That question can't be answered by eyeballing response times. It has to be measured, and measured correctly. Averages lie. Percentiles tell the truth. If you load-test your endpoint and report "average response time: 220ms," you have told your team almost nothing about what users actually experience. Averages get pulled toward the bulk of fast, easy requests and quietly absorb the slow ones. A service where 95% of requests return in 150ms and 5% take 4 seconds might average out to a very respectable-looking 330ms — while one user in twenty is sitting there watching a spinner for four full seconds. That's the tail, and the tail is what people remember and complain about. This is why the industry standard is to report p95 and p99 latency: the response time below which 95% (or 99%) of requests complete. p99 is especially important because at real scale, "rare" tail events happen constantly — if you serve a million requests a day, your p99 is happening ten thousand times. Throughput (requests per second, or for an LLM, tokens per second) is the companion metric: it tells you capacity, while percentiles tell you experience. You always read them together, because a system can hold a high RPS number while its p99 quietly balloons. LLM endpoints need LLM-specific metrics. A generic load-testing tool times a request from send to final byte received and calls that "latency." For a streaming LLM response, that single number hides the two things users actually feel: time-to-first-token (TTFT) — how long you stare at nothing before the response starts appearing — and inter-token latency (ITL), the gap between tokens once streaming starts, which determines whether text appears at a smooth reading pace or in stuttering bursts. A model could have a mediocre total-response-time number but excellent TTFT and smooth ITL, and feel snappy to a user. Measure total time alone and you'd never know that. Your capstone's load test needs to instrument TTFT and ITL separately, not just wall-clock total. The right-sizing method gives you a repeatable procedure instead of a guess:

The key insight in step 2 is that capacity limits reveal themselves as a knee in the curve — latency stays roughly flat as load increases, then bends sharply upward. That knee, not some arbitrary number, is your real ceiling. Tooling. This week's lab uses Locust, a Python-scripted, greenlet-based load-testing tool. You write a locustfile.py describing what a simulated user does, then drive it from Locust's web UI while watching RPS, latency, and failure rate update live. Locust is the right primary tool for the capstone because it's free, local, scriptable in the language you already know, and flexible enough to add custom TTFT/ITL timing around a streaming request. If your capstone exposes an OpenAI-compatible endpoint, the k6 + Grafana + InfluxDB "periscope" stack is a solid alternative worth knowing about, though Locust remains our primary tool for this course. A concrete target to reason against: for a user-facing API, a common SLO shape is p95 under 200ms and p99 under 500ms. Your own capstone SLO from Week 2 may differ — that's fine, use yours — but having any concrete numeric target, rather than a vague "should feel fast," is what turns load testing from a curiosity into an engineering decision. Correcting the misconceptions directly:

1. "Averages hide the tail" — judging performance by average response time actively conceals the tail-latency problems your real users hit. Always report and analyze p95/p99 together with throughput; never lead with an average.
2. "Optimize the actual bottleneck, not the layer you assume" — if you assume the fix is "add a GPU" or "add a replica" without profiling first, you'll often waste money. If GPU utilization sits below 50% during your load test, the problem is very likely inadequate batching, not insufficient hardware. And if your model is simply larger than your task requires, no infrastructure upgrade beats switching to a smaller, right-sized model. Profile before you provision.
3. "Unrealistic load tests give deceptively good numbers" — a Locust test that fires identical requests back-to-back with zero think-time, from the same machine on the same network as the server, will report throughput numbers that collapse the moment real users (with think-time, varied inputs, and real network latency) show up. Model realistic pacing and varied requests, or your "great numbers" are fiction.

By the end of this week you should be able to look at a latency-vs-load chart and immediately say where the knee is, what's causing it, and what you'd change — and then go make that change on your own deployment.