🛠️ Lab 12 — Benchmarking Tensor vs. Pipeline Parallelism on Dual GPUs (50 pts)

Goal: Use a free dual-GPU Kaggle notebook to launch vLLM with both tensor parallelism and pipeline parallelism, benchmark throughput and latency for each against your single-GPU baseline from Week 8, and explain the results in terms of inter-GPU communication. Steps:

  1. Open a new Kaggle Notebook and under Settings \> Accelerator, select GPU T4 x2 (free tier, 2x T4 GPUs). Enable internet access in the notebook settings.
  2. Install vLLM in the notebook environment: pip install vllm.
  3. Pick a mid-size open model that fits comfortably on 2x T4 GPUs in fp16 or AWQ quantized form (for example, a 7B-parameter open model). Confirm its attention-head count is evenly divisible by 2 so tensor-parallel-size 2 is valid.
  4. Launch the tensor-parallel deployment: vllm serve <your-model> --tensor-parallel-size 2. Once the server is up, send a batch of test prompts (use the same prompt set you used in your Week 8 single-GPU lab) and record throughput (tokens/sec) and per-request latency.
  5. Shut down the TP server, then launch the pipeline-parallel deployment: vllm serve <your-model> --pipeline-parallel-size 2. Run the identical prompt set and record the same throughput and latency metrics.
  6. Pull your Week 8 single-GPU benchmark numbers (or re-run that baseline in this notebook if needed) so you have three comparable result sets: single-GPU, TP-2, and PP-2.
  7. Build a comparison table (single-GPU vs. TP-2 vs. PP-2) showing throughput and average latency for each configuration.
  8. Write a short explanation (roughly 300–500 words) of your results that specifically addresses: why TP requires more inter-GPU communication (all-reduce synchronization after each parallelized layer) while PP only passes activations between stages; and how that difference shows up in your measured latency and throughput numbers.

Deliverables: Submit your notebook (.ipynb export or shared Kaggle link pasted as text), your comparison table, and your written explanation, either as an uploaded file or in the text entry box.