Title: Lab 13 — Queue-Aware Autoscaling with KEDA and the KAI Scheduler
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 13 — Autoscaling, Load Balancing, and the KAI Scheduler

🛠️ Lab 13 — Queue-Aware Autoscaling with KEDA and the KAI Scheduler (50 pts)

Goal: Stand up a local Kubernetes cluster running a vLLM-style inference deployment, wire up KEDA to autoscale it based on request queue depth (not CPU), then separately install the KAI Scheduler and demonstrate GPU sharing and gang scheduling — all using free, local tools. Steps:

1. Create a local cluster. Install kind (Kubernetes in Docker) if you don't already have it, and create a cluster: kind create cluster --name aiinfra201-w13.
2. Deploy Prometheus. Install the kube-prometheus-stack Helm chart into your cluster (free, open-source) so you have a Prometheus instance ready to scrape metrics.
3. Deploy your inference service. Reuse or redeploy your vLLM (or vLLM-compatible mock/simulator if you don't have GPU access locally) deployment from a previous week, exposing its built-in /metrics endpoint. Configure a Prometheus ServiceMonitor (or scrape config) to collect it.
4. Confirm the metric is visible. Port-forward to Prometheus and query for vLLM's queue-depth metric (e.g., vllm:num_requests_waiting) to confirm data is flowing before you wire up autoscaling.
5. Install KEDA. helm install keda kedacore/keda --namespace keda --create-namespace (free, open-source, no license required).
6. Write a KEDA ScaledObject. Create a ScaledObject manifest that targets your inference Deployment, uses the Prometheus scaler pointed at your queue-depth metric, and sets sensible minReplicaCount / maxReplicaCount and a target queue-depth-per-pod value.
7. Load-test and observe. Use a free load-testing tool (e.g., hey, k6, or a simple for loop with curl) to send a burst of requests. Watch kubectl get hpa,pods -w and record how many replicas KEDA spins up, and how long scale-in takes once the burst ends.
8. Install the KAI Scheduler (separate step, same or a second kind/minikube cluster). Install via Helm following the NVIDIA KAI Scheduler GitHub instructions (Apache 2.0, free). Define at least two scheduling queues (e.g., training and inference) with different priorities.
9. Demonstrate GPU sharing and gang scheduling. Following the K8s Recipes example referenced in KAI's documentation, deploy two or more simulated workloads that share a single (simulated) GPU device, and one multi-pod job that requires gang scheduling. Confirm in kubectl describe output or scheduler logs that pods are bin-packed and that the gang job either places all pods or none.
10. Write up your findings. In 1 paragraph, contrast what you saw HPA/KEDA do (replica count vs. queue depth) with what KAI did (pod placement vs. GPU sharing), and note one place in your setup where CPU-based scaling would have made the wrong decision.

Deliverables: Submit your KEDA ScaledObject YAML, your KAI queue/scheduling config YAML, a screenshot or terminal log showing scale-out and scale-in events, a screenshot or log showing GPU sharing/gang scheduling in action, and your written findings paragraph.