📖 Lecture — Scaling GPU Inference the Right Way

By now you've deployed inference servers on Kubernetes and you know how to request a GPU for a pod. This week we tackle a harder problem: keeping the right number of those pods running as traffic rises and falls, and routing requests to them intelligently once they exist. Get this wrong and you either waste expensive GPU capacity sitting idle, or you starve users with a growing request queue while your dashboards insist everything is fine. Why HPA breaks down for GPU inference. The Kubernetes Horizontal Pod Autoscaler (HPA) was built for a CPU/memory world: a web server under load burns more CPU, HPA sees the spike, and it adds replicas. GPU inference does not behave that way. A vLLM or Triton pod can sit at 15% CPU utilization while its GPU is completely saturated and its internal request queue is backing up by the second — CPU is just idling while the accelerator does the real work. If you point HPA at CPU or memory for a GPU-bound service, it will conclude nothing is wrong right up until users start timing out. This is the single most important idea this week: for GPU inference, the correct autoscaling signals are GPU utilization (e.g., dcgm_fi_dev_gpu_util from NVIDIA DCGM) and queue depth (from the engine's own metrics), not CPU. The production pattern is to combine those metrics with KEDA (Kubernetes Event-Driven Autoscaling), which scales workloads using external metrics rather than being limited to the built-in resource metrics HPA understands. KEDA in one paragraph. KEDA installs a ScaledObject custom resource that watches an external metrics source — in our case, Prometheus scraping vLLM's built-in /metrics endpoint — and translates that reading into a replica count, which it then feeds to the HPA machinery under the hood. You set a min and max replica bound, a target queue depth (say, 5 pending requests per pod), and KEDA handles the rest: scale-out when the queue grows, scale-in (often more cautiously, with a cooldown) when it drains. This is exactly what you'll build hands-on in the lab. Enter the KAI Scheduler. KEDA decides how many replicas you need, but something still has to decide where those pods land on a GPU cluster, especially when GPUs are shared or jobs must be placed together. That's the job of the KAI Scheduler — a high-performance Kubernetes scheduler that NVIDIA (via its Run:ai acquisition) open-sourced under Apache 2.0 in April 2025; it's now a CNCF Sandbox project. KAI was purpose-built for the full AI lifecycle, from interactive notebooks to large distributed training and inference, and it solves three problems the default Kubernetes scheduler does not:

KAI is also compatible with node-level autoscalers like Karpenter, so pod-level scheduling decisions (KAI) and node-level infrastructure scaling (Karpenter) work together rather than fighting each other. Load balancing in front of your replicas. Once you have a healthy set of replica pods, a Kubernetes Ingress or an NGINX load balancer decides which pod gets the next request. Two common strategies:

• Round-robin — requests are distributed evenly in sequence, regardless of what each pod is currently doing. Simple, but blind to load: a pod already juggling three long-running generation requests can still get the next one.
• Least-connections — requests go to whichever pod currently has the fewest active connections. Better for inference, since generation requests vary wildly in duration, but it's still a proxy for load, not queue depth itself.

Neither strategy "understands" GPU queue depth the way KEDA's metrics pipeline does — this is an active area of the field, and part of why teams increasingly push scaling logic down to queue-aware metrics rather than relying on the load balancer alone. Correcting three common misconceptions:

1. "The Kubernetes Cluster Autoscaler sees all GPU resource requests the same way." Not true. GPU-sharing tools frequently place fractional GPU requests in pod annotations rather than the standard resources.requests field. The Cluster Autoscaler only reads resources.requests — it never inspects annotations — so a pod requesting a fractional GPU via annotation can fail to trigger node scale-up even though it's technically unschedulable.
2. "Any Kubernetes scheduler can correctly place a multi-pod distributed job." Not true. Ordinary scheduling treats every pod as an independent decision. A distributed training or inference job with, say, 4 worker pods needs all 4 placed before any of them are useful — that's what gang scheduling (KAI) provides, and the default scheduler has no concept of it.
3. "Ordinary HPA based on CPU/memory works fine for GPU inference." Not true, as covered above — GPU utilization and queue depth are the correct signals; CPU stays misleadingly low during GPU-bound work.

By the end of this week, you'll have configured both halves of this story yourself: a KEDA ScaledObject scaling on real queue-depth metrics, and a KAI Scheduler managing GPU sharing and gang placement underneath it.