📖 Lecture — From Container to Cluster: Serving Models on Kubernetes

Last week you learned to package an application in a container. This week you'll take that same skill and point it at a real job: serving an AI model to users, reliably, on Kubernetes. The good news is that the core pattern is simpler than it sounds, and it rests on two objects you'll use constantly for the rest of this course: the Deployment and the Service. A minimal AI inference app on Kubernetes pairs a Deployment — which runs your containerized model-serving code — with a Service, which exposes a stable network endpoint in front of it. Inside each Pod that the Deployment manages, you're running some kind of model-serving process: it might be a hand-written FastAPI app that loads a scikit-learn or PyTorch model and exposes a /predict route, or it might be a purpose-built inference server like vLLM or NVIDIA Triton Inference Server. The Deployment's job is to keep a specified number of Pod replicas running, restart them if they crash, and roll out updates without downtime. The Service's job is to give all of those (potentially many, potentially changing) Pods one stable DNS name and IP address that other things — including your own curl command in Lab 8 — can talk to, load-balancing requests across whichever replicas are currently healthy. Why do you need a whole separate object just for networking? Because Pods are disposable. Kubernetes can kill and reschedule a Pod at any time, and each new Pod gets a new internal IP address. If your client code had to track individual Pod IPs, it would break constantly. The Service solves this by sitting in front of the Deployment and routing traffic by label selector, not by IP — so as Pods come and go, the Service just keeps working. As your serving needs grow more sophisticated than "one Deployment, one Service," you'll want a layer that manages scaling, traffic splitting, and rollout strategy for you. That's where KServe comes in. KServe is a Kubernetes-native orchestration layer for model serving built around a custom resource called the InferenceService. You describe what model you want served and where it lives, and KServe handles autoscaling (including scaling to zero when idle), canary rollouts between model versions, and request routing — while delegating the actual number-crunching to a backend runtime underneath it, such as Triton or vLLM. It helps to think of these three tools as sitting at different altitudes:

vLLM and Triton are both inference servers, but they're optimized for different jobs. vLLM specializes in high-throughput LLM serving, using a memory-management technique called PagedAttention to pack more concurrent requests into limited GPU memory. Triton is more general-purpose: it can serve models from multiple frameworks (PyTorch, ONNX, TensorFlow) side by side behind one server. KServe doesn't replace either of these — it sits above them, deciding how many replicas to run and how to route traffic, while Triton or vLLM does the actual inference math. Finally, you don't always have to write any of this by hand. Cloud-managed AI toolchains — Azure KAITO and Google GKE's LLM deployment operator among them — exist precisely to simplify deploying and managing open-source LLMs on Kubernetes. Instead of authoring Deployment, Service, and InferenceService manifests yourself, you describe the model you want (often by name) and the managed operator generates and manages the underlying resources. These tools are worth knowing about even though this week's lab has you do it the manual way first — understanding the manifests underneath is what makes the managed abstractions trustworthy rather than mysterious. Before you get to the lab, let's clear up two things people new to GPU workloads often get wrong. Misconception 1: "High nvidia-smi GPU utilization means my inference is healthy and efficient." Not necessarily. nvidia-smi reporting near-100% utilization can just as easily mean thrashing — the GPU is busy, but inefficiently, because batch sizes are poorly tuned or multiple workloads are contending for the same memory bandwidth. A GPU stuck swapping data in and out of memory because batches are too small (or too many jobs are packed on) can show high utilization while throughput is actually poor. Always pair utilization numbers with throughput and latency metrics before concluding a workload is running well. Misconception 2: "Kubernetes automatically places multi-GPU workloads on GPUs that are close together." It does not, by default. The default Kubernetes scheduler has no awareness of GPU interconnect topology — it doesn't know or care whether two GPUs are connected via a fast NVLink or sit in entirely different physical servers connected only by a slower network fabric. For single-GPU inference (which is what you'll do in Lab 8), this doesn't matter. But for multi-GPU training or serving jobs that need fast GPU-to-GPU communication, the scheduler can innocently place your Pods on GPUs (or nodes) that make the workload slow or even infeasible — you need topology-aware scheduling extensions to fix this, a topic we'll return to in later weeks. With that grounding, you're ready to build and deploy your own small inference service end to end in Lab 8.