Title: AIINFRA 200: Production Inference Serving & GPU Orchestration

00 · Course Information
  Outcomes & Rubrics
Week 01 — Linux Server Foundations for AI Workloads
  Lecture 1 — From Bare Metal to Model-Ready: Linux and GPU Server Fundamentals
  Lab 1 — Systemd-Managed GPU Inference Process on a Free-Tier Cloud VM
  Discussion 1 — systemd vs. Containers and Orchestrators
  Quiz 1 — Linux Server Foundations for AI Workloads
  Week 01 Resources
Week 02 — Docker, GPU Runtime, and Dev-Only Tools (Ollama, LM Studio)
  Lecture 2 — Getting GPUs Into Containers, and Where Dev Tools Fit
  Lab 2 — GPU Passthrough and a Local Ollama + Open WebUI Stack
  Discussion 2 — Containerize It or Keep It on the Host?
  Quiz 2 — Docker, GPU Runtime, and Dev-Only Tools (Ollama, LM Studio)
  Week 02 Resources
Week 03 — Production Inference Fundamentals and the vLLM Engine
  Lecture 3 — Production Inference Fundamentals and the vLLM Engine
  Lab 3 — Standing Up a Production-Style vLLM Server
  Discussion 3 — Why Did Every Serving Engine Converge on the Same Architecture?
  Quiz 3 — Production Inference Fundamentals and the vLLM Engine
  Week 03 Resources
Week 04 — PagedAttention and Continuous Batching Deep Dive
  Lecture 4 — Inside vLLM's Memory and Scheduling Engine
  Lab 4 — Measuring Prefix Caching and Concurrency Gains in vLLM
  Discussion 4 — Rethinking the Throughput/Latency Tradeoff
  Quiz 4 — PagedAttention and Continuous Batching Deep Dive
  Week 04 Resources
Week 05 — Hugging Face TGI and SGLang Serving Engines
  Lecture 5 — From TGI to SGLang: The New Serving Engine Landscape
  Lab 5 — TGI vs. SGLang Head-to-Head Benchmark
  Discussion 5 — Why Did TGI Lose Its Edge?
  Quiz 5 — Hugging Face TGI and SGLang Serving Engines
  Week 05 Resources
Week 06 — TensorRT-LLM Compilation and Optimization
  Lecture 6 — Compiling Models with TensorRT-LLM: From Checkpoint to Optimized Engine
  Lab 6 — Quantize and Compile TinyLlama with TensorRT-LLM
  Discussion 6 — Compile-Ahead vs. Serve-and-Adapt
  Quiz 6 — TensorRT-LLM Compilation and Optimization
  Week 06 Resources
Week 07 — NVIDIA Triton Inference Server and Model Repositories
  Lecture 7 — NVIDIA Triton Inference Server and Model Repositories
  Lab 7 — Standing Up a Local Triton Model Repository
  Discussion 7 — Multi-Backend Servers vs. Single-Engine Serving
  Quiz 7 — NVIDIA Triton Inference Server and Model Repositories
  Week 07 Resources
Week 08 — Serving Engine Benchmarking Lab
  Lecture 8 — Benchmarking Serving Engines the Right Way
  Midterm Exam
  Discussion 8 — Whose Benchmark Do You Trust?
  Week 08 Resources
Week 09 — Ray Serve and KServe for Scalable Model Serving
  Lecture 9 — Two Roads to Scalable Model Serving: Ray Serve and KServe
  Lab 9 — Deploying and Scaling a Model with Ray Serve and KubeRay
  Discussion 9 — Ray Serve or KServe: What Would You Recommend?
  Quiz 9 — Ray Serve and KServe for Scalable Model Serving
  Week 09 Resources
Week 10 — GPU Orchestration on Kubernetes with the NVIDIA GPU Operator
  Lecture 10 — From Manual GPU Setup to the Kubernetes Operator Pattern
  Lab 10 — Deploying the NVIDIA GPU Operator on Minikube
  Discussion 10 — Operator Automation vs. Manual GPU Node Management
  Quiz 10 — GPU Orchestration on Kubernetes with the NVIDIA GPU Operator
  Week 10 Resources
Week 11 — Dynamic Resource Allocation and MIG Partitioning
  Lecture 11 — From Device Plugins to Dynamic Resource Allocation and MIG
  Lab 11 — MIG Partition Planning and DRA Claim Design
  Discussion 11 — MIG + DRA vs. "One Pod, One GPU"
  Quiz 11 — Dynamic Resource Allocation and MIG Partitioning
  Week 11 Resources
Week 12 — Multi-GPU Tensor and Pipeline Parallelism
  Lecture 12 — Splitting the Model: Tensor vs. Pipeline Parallelism
  Lab 12 — Benchmarking Tensor vs. Pipeline Parallelism on Dual GPUs
  Discussion 12 — Choosing TP vs. PP for Your Deployment
  Quiz 12 — Multi-GPU Tensor and Pipeline Parallelism
  Week 12 Resources
Week 13 — Autoscaling, Load Balancing, and the KAI Scheduler
  Lecture 13 — Scaling GPU Inference the Right Way
  Lab 13 — Queue-Aware Autoscaling with KEDA and the KAI Scheduler
  Discussion 13 — CPU-Bound vs. GPU-Bound Observability
  Quiz 13 — Autoscaling, Load Balancing, and the KAI Scheduler
  Week 13 Resources
Week 14 — Disaggregated Prefill/Decode with NVIDIA Dynamo
  Lecture 14 — Splitting Prefill and Decode: Why and How Dynamo Does It
  Lab 14 — Disaggregated vs. Aggregated vLLM with NVIDIA Dynamo
  Discussion 14 — When Does Disaggregation Actually Pay Off?
  Quiz 14 — Disaggregated Prefill/Decode with NVIDIA Dynamo
  Week 14 Resources
Week 15 — Observability, Cost Analysis, and Right-Sizing
  Lecture 15 — Seeing What Your GPUs Are Actually Doing (and What It Costs You)
  Lab 15 — GPU Observability Stack and Cost-Per-1000-Tokens Model
  Discussion 15 — Why Are GPUs So Often Underutilized?
  Quiz 15 — Observability, Cost Analysis, and Right-Sizing
  Week 15 Resources
Week 16 — Capstone Project & Course Review
  Final Capstone Project
  Capstone Presentation
  Course Review & Reflection
  What's Next