Title: Lecture 14 — Splitting Prefill and Decode: Why and How Dynamo Does It
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 14 — Disaggregated Prefill/Decode with NVIDIA Dynamo

📖 Lecture — Splitting Prefill and Decode: Why and How Dynamo Does It

By now you've stood up single-GPU and co-located multi-GPU inference deployments in earlier weeks. This week we look at a pattern used at datacenter scale: splitting the two phases of autoregressive generation — prefill and decode — onto separate pools of GPUs, and using NVIDIA Dynamo to coordinate the result.

Two phases, two very different jobs

Every LLM request runs through two distinct phases. Prefill processes the entire input prompt in parallel to produce the first output token. Because every prompt token can be processed simultaneously, prefill is compute-bound: its cost scales with prompt length, and it benefits from large batches and low tensor parallelism, since low parallelism reduces the cross-GPU communication overhead that would otherwise eat into throughput. Decode generates the remaining tokens one at a time, each new token depending on the last. Decode is memory-bandwidth-bound and latency-sensitive — the GPU spends most of its time reading the growing KV cache from memory rather than doing dense compute, and it benefits from higher tensor parallelism to keep per-token latency low. Because these two phases want opposite things (low vs. high parallelism, throughput-optimized vs. latency-optimized), running them on the same GPU forces a compromise. Disaggregation — giving prefill and decode their own separately sized and separately scaled GPU pools — lets each phase be resourced for what it actually needs.

Phase | Bottleneck | Scales with | Wants
Prefill | Compute | Prompt length | Large batches, low tensor parallelism
Decode | Memory bandwidth | Output length / concurrent sequences | High tensor parallelism, low latency

Where Dynamo fits

NVIDIA Dynamo (open source, ai-dynamo/dynamo on GitHub, announced at GTC 2025) is a datacenter-scale inference orchestration framework. It does not replace your inference engine — it sits above engines like vLLM, TensorRT-LLM, and SGLang, all of which have full feature parity within Dynamo for disaggregated serving, KV-aware routing, and multimodal workloads. Dynamo's job is to coordinate many GPUs and nodes into one inference system: deciding which worker handles a request, moving KV-cache data between GPUs, scaling pools up and down, and recovering from failures. Two Dynamo components matter most for this week:

- KV-aware Router — directs each incoming request to the worker already holding the most relevant cached KV state, avoiding redundant recomputation of prompt prefixes that other requests have already processed.
- NIXL — a low-latency communication library that accelerates the transfer of KV-cache data between the prefill and decode GPU pools. NVIDIA reports this combination boosting served requests up to 30x for DeepSeek-R1 on Blackwell hardware in favorable conditions.

You can install Dynamo with pip install ai-dynamo[all] inside a Python virtual environment, or pull a prebuilt runtime container if you'd rather not manage the dependency chain by hand.

Correcting three common misconceptions

"Dynamo is basically a switch that turns on prefill/decode separation." Not quite. Splitting the two phases across GPU pools is a deployment topology choice — Dynamo's actual value is everything required to make that topology work in production: request routing, GPU-to-GPU KV-cache transfer, worker placement, autoscaling, service discovery, and fault handling. Without that coordination layer, disaggregation is just two disconnected GPU pools. "Disaggregating is always a performance win." It isn't. For many standard chatbot workloads or moderate-scale RAG applications, aggregated serving (prefill and decode co-located on the same GPU) remains simpler to operate and often performs just as well or better — there's no cross-pool KV transfer to pay for, and utilization stays high without careful load balancing between two pools. Dynamo keeps aggregated topology as a first-class, fully supported deployment mode precisely because it's frequently the right choice. Disaggregation earns its complexity at scale, with long prompts, high concurrency, or strict latency SLAs on token generation. "Prefill and decode are basically the same workload, just different token counts." They are not interchangeable. Prefill's cost is driven by prompt length and is compute-bound; decode's cost is driven by how many tokens you're generating for how many concurrent sequences, and it's bound by how fast the GPU can read cached state from memory. A GPU pool tuned for one will typically be poorly matched to the other, which is exactly why separating them — and resourcing each independently — is the whole point of disaggregation.

Takeaway

Disaggregated prefill/decode is a tool for a specific problem: mismatched resource profiles at scale. NVIDIA Dynamo makes the tool practical by handling the orchestration work that disaggregation demands, while still letting you run the simpler aggregated topology when that's the better fit.