Title: Lecture 12 — Splitting the Model: Tensor vs. Pipeline Parallelism
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 12 — Multi-GPU Tensor and Pipeline Parallelism

📖 Lecture — Splitting the Model: Tensor vs. Pipeline Parallelism

By now you've deployed a large language model on a single GPU. But what happens when the model doesn't fit on one GPU, or when one GPU simply can't push enough throughput for your traffic? The answer is multi-GPU serving, and this week we cover the two foundational strategies for splitting a model across GPUs: tensor parallelism and pipeline parallelism.

Tensor parallelism: splitting the math inside a layer

Tensor parallelism (TP), sometimes called intra-layer parallelism, cuts individual tensor operations and weight matrices into chunks and hands each chunk to a different GPU. For example, a single attention or feed-forward weight matrix might be split column-wise across 4 GPUs, each GPU computing a partial result. Those partial results then have to be synchronized — typically via an all-reduce operation — before the layer's output can move forward. This synchronization happens after every parallelized layer, which means TP generates frequent, latency-sensitive, high-bandwidth communication between GPUs. That's exactly why TP is best suited to GPUs connected by NVLink or another high-bandwidth interconnect: the constant chatter between GPUs would bottleneck badly over a slower link like PCIe.

Pipeline parallelism: splitting the model into stages

Pipeline parallelism (PP), or inter-layer parallelism, takes a completely different cut. Instead of slicing individual weight matrices, PP assigns contiguous ranges of whole layers to different GPUs — GPU 0 might hold layers 1–8, GPU 1 holds layers 9–16, and so on. Each GPU runs its full stage of layers and then hands off only the activations (the intermediate output tensor) to the next GPU in the pipeline. There's no all-reduce, no weight synchronization — just a single tensor passed downstream. That makes PP far more tolerant of slower interconnects like PCIe, and it's the better choice when you're prioritizing overall throughput over the latency of any single request.

Comparing the two strategies

Dimension | Tensor Parallelism (TP) | Pipeline Parallelism (PP)
What's split | Individual weight matrices / tensor ops within a layer | Contiguous ranges of whole layers
Communication pattern | Frequent all-reduce sync after each parallelized layer | Single activation hand-off between stages
Bandwidth needs | High — needs NVLink or similar | Lower — tolerates PCIe
Best for | Low request rates, latency-sensitive workloads | High-throughput demand
Effect on latency | Lowers per-token latency (at communication cost) | Can add pipeline "bubble" latency but scales throughput well
Typical topology | Within a single node (GPUs sharing NVLink) | Across nodes

Configuring it in vLLM

On a single node with 4 GPUs sharing NVLink, you can launch tensor-parallel serving with:

vllm serve facebook/opt-13b --tensor-parallel-size 4

A critical constraint: the TP size must be a power of 2, and it must evenly divide the model's attention-head count — vLLM will error out otherwise, so always check the model config first. When you have more GPUs spread across multiple nodes, you can combine both strategies. For example, across 8 GPUs (2 nodes of 4):

vllm serve facebook/opt-13b --tensor-parallel-size 4 --pipeline-parallel-size 2

The production pattern: combine, don't choose

A very common real-world deployment pattern sets tensor-parallel size equal to the number of GPUs per node (since those GPUs share fast NVLink) and pipeline-parallel size equal to the number of nodes (since inter-node links are typically slower Ethernet or PCIe-class bandwidth). This gets you the latency benefits of TP where communication is cheap, and the throughput benefits of PP where communication is expensive.

Correcting the common misconceptions

1. "TP and PP are interchangeable multi-GPU modes." They are not. TP splits individual layer weights and requires frequent synchronization; PP splits whole layers and only passes activations between stages. Picking the wrong one for your hardware can tank performance.
2. "You must choose either TP or PP." In production, you typically use both simultaneously — TP within a node, PP across nodes — rather than picking one exclusively.
3. "The communication patterns are basically the same." They are not: TP demands high-bandwidth all-reduce synchronization after every layer, while PP only sends a single, lower-bandwidth activation hand-off between stages. This difference is exactly why interconnect type (NVLink vs. PCIe) drives which strategy you choose.

Understanding this distinction — what gets split, how much communication it generates, and what hardware it demands — is the foundation for every multi-GPU serving decision you'll make going forward.