In Week 2, you stood up Ollama on your own machine and talked to a local model. That was a great way to learn how inference works end to end, but Ollama is fundamentally a development tool — built to make one model respond reasonably fast to one user at a time on a laptop or workstation GPU. Production inference is a different problem: dozens or thousands of concurrent users, unpredictable prompt lengths, and a hard requirement to keep expensive GPUs busy without falling over. This week we meet the tool that the industry has largely standardized on for that problem: vLLM.
vLLM is an open-source inference-serving library that came out of UC Berkeley's Sky Computing Lab. It is not a new model architecture or a new training technique — it's a serving engine: software that sits between a trained model's weights and the network requests asking that model to generate text. vLLM supports 200+ model architectures (Llama, Qwen, Mistral, Gemma, and many more) and exposes an OpenAI-compatible API server, meaning any tool or script written against OpenAI's /v1/chat/completions or /v1/completions endpoints can point at a vLLM server instead, often with just a URL change. The technique vLLM is best known for is PagedAttention, introduced in the SOSP 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention." It's worth being precise about what this actually solves, because it's the single most misunderstood idea in this space (more on that below).
Every autoregressive transformer keeps a KV-cache — the accumulated key and value tensors for every token generated so far — so it doesn't have to recompute attention over the whole sequence at each new step. The problem: naive serving engines allocate one large, contiguous chunk of GPU memory per request, sized for the maximum possible sequence length. Most requests don't use anywhere near that much, so huge amounts of GPU memory sit reserved-but-unused. Worse, as requests of different lengths start and finish, memory gets fragmented into unusable gaps, the same way a disk fragments over years of file creation and deletion. PagedAttention borrows the fix directly from operating-system virtual memory: instead of one contiguous block per sequence, the KV-cache is split into fixed-size blocks (analogous to memory pages), and a per-request block table maps logical token positions to physical blocks that can live anywhere in GPU memory. Blocks are allocated on demand, released immediately when no longer needed, and can even be shared between sequences (for example, across the parallel samples of a single prompt). The result is dramatically less wasted memory and near-zero fragmentation — which translates directly into more concurrent requests fitting on the same GPU.
PagedAttention manages memory. A separate mechanism, continuous batching (also called iteration-level scheduling), manages scheduling. Older serving systems used static batching: a batch of requests starts together and the whole batch waits until every request in it finishes before a new batch can begin — so a single long-running request blocks GPU capacity that could serve five short ones. Continuous batching instead evaluates the batch at every generation step (every token), immediately swapping a finished sequence out and pulling a new, waiting request in. The GPU is never idle waiting for the slowest member of a batch to finish.
Since v0.6.0, vLLM's V1 engine is the default and refines this further. Two changes matter most for this week:
A single GPU can't hold every model, and even when it can, a single request stream can't saturate a datacenter. vLLM supports five parallelism strategies you should be able to name and distinguish:
| Parallelism type | What's split across devices | Typical use case |
|---|---|---|
| Tensor parallel (TP) | Individual weight matrices, within a layer | Model too large for one GPU's memory |
| Pipeline parallel (PP) | Different layers/stages of the model | Very large models across multiple nodes |
| Data parallel (DP) | Full model replicas, different requests | Scaling raw request throughput |
| Expert parallel (EP) | Different experts in a Mixture-of-Experts model | Serving MoE models (e.g., Mixtral-style) efficiently |
| Context parallel (CP) | The sequence/context dimension itself | Extremely long context windows |
For this week's lab you'll run entirely on a single free-tier GPU, so none of this parallelism is required — but you should recognize that the exact same vllm serve command you'll use is the on-ramp to multi-GPU, multi-node production deployments. You don't switch tools as you scale; you add flags.
Everything above is packaged behind a deceptively simple command:
vllm serve <model-id>This single command downloads (or loads) the model, wraps it in PagedAttention-managed KV-cache, enables continuous batching, configures tensor/pipeline parallelism automatically if you request multiple GPUs, and exposes an OpenAI-compatible HTTP API on a local port. That's the entire leap from Week 2's single-user Ollama chat to a real, networked inference service — which is exactly what you'll build in this week's lab.
Competing serving engines — Hugging Face's TGI, SGLang, NVIDIA's TensorRT-LLM — each have their own strengths, but all of them converged on the same core ideas vLLM popularized: paged/block-based KV-cache management, continuous batching, and multi-GPU parallelism. When an entire industry independently arrives at the same architecture, that's a strong signal the architecture is solving a real, fundamental constraint (GPU memory) rather than a fashion. That's why understanding vLLM deeply gives you a template for reasoning about any production inference engine you'll encounter on the job.
"Bigger batches are always better." Not true. Throughput (tokens/sec across all requests) does generally rise as you admit more concurrent requests, but tail latency — how long the slowest, unluckiest request waits — rises too, and beyond a certain batch size it rises sharply. For an interactive chat application, a user who waits 8 seconds for a reply because the server was busy maximizing throughput has a bad experience, even if the server's aggregate numbers look great. Production tuning is a deliberate trade-off, not a "max out the batch size" exercise. **"Throughput is the metric." There is no single number that defines a "good" serving engine — it depends entirely on the use case. An interactive chat product cares most about TTFT (time-to-first-token) and per-token latency, and will intentionally cap batch size to protect that experience even at the cost of some throughput. A nightly batch-summarization job over millions of documents cares about tokens/sec** and total completion time, and will happily accept higher latency per request in exchange for maximum GPU utilization. Know which metric matters before you tune anything. "vLLM's whole purpose is batching requests together." Batching (continuous batching, specifically) is one of two pillars, but PagedAttention is not a batching technique at all — it is a memory-management technique, conceptually borrowed from OS virtual memory, that solves KV-cache fragmentation and waste. You could imagine a system with continuous batching but naive memory allocation (it would run out of GPU memory constantly), or a system with perfect memory management but static batching (it would waste GPU cycles). vLLM's achievement is combining both, well, in one engine.