📖 Lecture — From Manual GPU Setup to the Kubernetes Operator Pattern

In AIINFRA 101 you learned Kubernetes fundamentals on CPU-only workloads. The moment you add GPUs to the picture, a new problem shows up: every worker node needs a matching NVIDIA driver, a container runtime configured to expose the GPU, a device plugin so the scheduler knows the GPU exists, and some way to monitor it. Doing that by hand, node by node, does not scale — and it drifts out of sync the moment you add a node, upgrade a driver, or autoscale a GPU node pool. This week's core idea is that the NVIDIA GPU Operator solves this by applying the Kubernetes operator pattern: instead of you SSH-ing into nodes and running install scripts, you describe the desired GPU state declaratively, and a set of controllers running inside the cluster continuously reconcile every node toward that state.

What the operator pattern buys you

An "operator" in Kubernetes is just a custom controller that manages an application's full lifecycle — install, configure, upgrade, monitor — using Kubernetes-native objects instead of imperative scripts. The GPU Operator packages five previously-manual jobs into containerized components, most of them deployed as DaemonSets (one pod per matching node, automatically):

Because these run as DaemonSets, adding a new GPU node to the cluster is enough to trigger the whole stack to deploy on it automatically — no manual per-node setup step required.

Installing it

The Operator is distributed as a Helm chart. A default install looks like this:

helm repo add nvidia https://nvidia.github.io/gpu-operator
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version=v26.3.3

That single command deploys the driver, Container Toolkit, device plugin, DCGM Exporter, and MIG Manager on every GPU worker node the cluster has — a declarative, one-line replacement for what used to be a per-node runbook.

How the scheduler actually sees a GPU

The Kubernetes device plugin framework is the mechanism that lets the scheduler treat a GPU as a first-class resource, the same way it treats CPU or memory. Once the device plugin is running, a node advertises something like nvidia.com/gpu: 1 in its allocatable resources, and you request it in a pod spec exactly like any other resource:

resources:
  limits:
    nvidia.com/gpu: 1

The device plugin's job stops at advertising and allocating the resource to a pod. The Container Toolkit is what actually does the work of wiring up the real device files, driver libraries, and runtime hooks inside the container once that pod lands on a node — two distinct jobs, done by two distinct components, and it is worth keeping them separate in your mental model.

The distinctive (and slightly odd) part: installing host kernel modules from a container

Here is the piece that trips almost everyone up the first time: the driver DaemonSet's pod is a container — yet the job of that container is to install the NVIDIA kernel module directly onto the host operating system, not into itself. This is an unusual operational pattern compared to typical containerized workloads, which are supposed to be isolated from the host. The GPU Operator's driver container is deliberately privileged and mounts host paths so it can compile/load the kernel module on the node it is running on. Understanding this is the key to understanding almost every subsequent misconception about the Operator.

Correcting three common misconceptions

1. "Installing the GPU Operator puts nvidia-smi on the host machine." Not true. nvidia-smi is installed and runs inside the driver container/pod the Operator deploys — not directly on the host's shell. If you want to run it yourself, you must kubectl exec into that driver pod (typically named nvidia-driver-daemonset-* in the gpu-operator namespace) rather than expecting it on the node's own command line.
2. "The plain NVIDIA device plugin and the full GPU Operator require the same amount of setup." Not true. The device plugin only handles resource advertisement and scheduling — you would still have to manually install the correct driver version and configure the container runtime yourself first. The GPU Operator automates all of that surrounding setup too, which is the entire point of adopting it.
3. "Docker is the expected container runtime for GPU Operator setups." This was true historically and still appears in some older tutorials, but most modern managed Kubernetes services no longer support Docker as the node runtime at all. In practice today, containerd with the NVIDIA Container Toolkit configured for containerd is the realistic default, and that is what you should expect to configure in this week's lab.

By the end of this week, you should be able to look at kubectl get pods -n gpu-operator and kubectl describe node output and name what each running component is doing for you — that diagnostic fluency is exactly what the lab is designed to build.