Title: Quiz 12 — Multi-GPU Tensor and Pipeline Parallelism
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 12 — Multi-GPU Tensor and Pipeline Parallelism

Q: What does tensor parallelism split across GPUs?
- Whole contiguous ranges of model layers
- Individual tensor operations and weight matrices within a layer
- The KV cache only
- The tokenizer vocabulary
A: Individual tensor operations and weight matrices within a layer

Q: What does pipeline parallelism split across GPUs?
- Individual weight matrices
- Contiguous ranges of model layers, assigned to different GPUs
- The attention heads of a single layer only
- The batch dimension only
A: Contiguous ranges of model layers, assigned to different GPUs

Q: Tensor parallelism requires frequent all-reduce synchronization between GPUs after each parallelized layer.
- True
- False
A: True

Q: Pipeline parallelism and tensor parallelism are interchangeable — you should always pick one and never combine them.
- True
- False
A: False

Q: Which interconnect is best suited to tensor parallelism, given its frequent high-bandwidth synchronization needs?
- PCIe only
- NVLink or another high-bandwidth interconnect
- Standard Ethernet
- USB-C
A: NVLink or another high-bandwidth interconnect

Q: In vLLM, what is the command-line flag used to set the tensor-parallel size, and what numeric constraint must its value satisfy relative to the model's attention-head count?
Feedback: Sample/expected answer: `--tensor-parallel-size`****; its value must be a power of 2 that evenly divides the model's attention-head count

Q: In common production practice combining both strategies across multiple nodes, how are TP size and PP size typically set?
- TP size = number of nodes; PP size = GPUs per node
- TP size = GPUs per node; PP size = number of nodes
- Both are always set to the total GPU count
- TP size is always 1
A: TP size = GPUs per node; PP size = number of nodes

Q: Which workload characteristic makes pipeline parallelism the more attractive choice over tensor parallelism?
- A need to minimize per-token latency for a single request at low request rates
- High-throughput demand where lower cross-GPU communication overhead matters more than single-request latency
- A model that fits entirely on one GPU
- A requirement to avoid using any interconnect at all
A: High-throughput demand where lower cross-GPU communication overhead matters more than single-request latency