By now you've deployed a large language model on a single GPU. But what happens when the model doesn't fit on one GPU, or when one GPU simply can't push enough throughput for your traffic? The answer is multi-GPU serving, and this week we cover the two foundational strategies for splitting a model across GPUs: tensor parallelism and pipeline parallelism.
Tensor parallelism (TP), sometimes called intra-layer parallelism, cuts individual tensor operations and weight matrices into chunks and hands each chunk to a different GPU. For example, a single attention or feed-forward weight matrix might be split column-wise across 4 GPUs, each GPU computing a partial result. Those partial results then have to be synchronized — typically via an all-reduce operation — before the layer's output can move forward. This synchronization happens after every parallelized layer, which means TP generates frequent, latency-sensitive, high-bandwidth communication between GPUs. That's exactly why TP is best suited to GPUs connected by NVLink or another high-bandwidth interconnect: the constant chatter between GPUs would bottleneck badly over a slower link like PCIe.
Pipeline parallelism (PP), or inter-layer parallelism, takes a completely different cut. Instead of slicing individual weight matrices, PP assigns contiguous ranges of whole layers to different GPUs — GPU 0 might hold layers 1–8, GPU 1 holds layers 9–16, and so on. Each GPU runs its full stage of layers and then hands off only the activations (the intermediate output tensor) to the next GPU in the pipeline. There's no all-reduce, no weight synchronization — just a single tensor passed downstream. That makes PP far more tolerant of slower interconnects like PCIe, and it's the better choice when you're prioritizing overall throughput over the latency of any single request.
| Dimension | Tensor Parallelism (TP) | Pipeline Parallelism (PP) |
|---|---|---|
| What's split | Individual weight matrices / tensor ops within a layer | Contiguous ranges of whole layers |
| Communication pattern | Frequent all-reduce sync after each parallelized layer | Single activation hand-off between stages |
| Bandwidth needs | High — needs NVLink or similar | Lower — tolerates PCIe |
| Best for | Low request rates, latency-sensitive workloads | High-throughput demand |
| Effect on latency | Lowers per-token latency (at communication cost) | Can add pipeline "bubble" latency but scales throughput well |
| Typical topology | Within a single node (GPUs sharing NVLink) | Across nodes |
On a single node with 4 GPUs sharing NVLink, you can launch tensor-parallel serving with:
vllm serve facebook/opt-13b --tensor-parallel-size 4A critical constraint: the TP size must be a power of 2, and it must evenly divide the model's attention-head count — vLLM will error out otherwise, so always check the model config first. When you have more GPUs spread across multiple nodes, you can combine both strategies. For example, across 8 GPUs (2 nodes of 4):
vllm serve facebook/opt-13b --tensor-parallel-size 4 --pipeline-parallel-size 2A very common real-world deployment pattern sets tensor-parallel size equal to the number of GPUs per node (since those GPUs share fast NVLink) and pipeline-parallel size equal to the number of nodes (since inter-node links are typically slower Ethernet or PCIe-class bandwidth). This gets you the latency benefits of TP where communication is cheap, and the throughput benefits of PP where communication is expensive.
Understanding this distinction — what gets split, how much communication it generates, and what hardware it demands — is the foundation for every multi-GPU serving decision you'll make going forward.