So far in this course, every container you've built has run on the CPU. That's fine for a web server or a batch script, but most of the AI workloads this certificate is preparing you for — model training, large-batch inference, computer vision pipelines — need a GPU. This week we answer a deceptively simple question: how does a container, which is supposed to be an isolated, portable little sandbox, reach out and use a physical GPU sitting on the host machine? The answer is the NVIDIA Container Toolkit (currently at version 1.19.1). It's not a driver, and it's not CUDA itself — it's a suite of tools that configures the Docker daemon so that containers can be granted access to the host's NVIDIA GPU driver at the moment they run. Once configured, a container can call nvidia-smi, load CUDA libraries, and run GPU compute — all without the toolkit itself being installed inside the container. The toolkit lives on the host, mediating between Docker and the driver. Why does it live on the host instead of in the image? Because the GPU driver is tightly coupled to the host's kernel and hardware. If every image had to bundle its own driver, you'd need a different image for every host configuration, and driver mismatches would break things constantly. Instead, the host owns the driver, and the toolkit exposes it to whichever container asks for it at run time.
Installation has a specific order of operations, and it's worth memorizing the shape of it rather than just the commands:
| Step | What you do | Why |
|---|---|---|
| 1 | Install the host's NVIDIA GPU driver (not the full CUDA Toolkit) | The driver is the only host-side dependency; CUDA itself can live inside containers |
| 2 | Add the NVIDIA Container Toolkit's apt repository |
Gives you access to versioned toolkit packages |
| 3 | apt-get install nvidia-container-toolkit |
Installs the toolkit binaries, including nvidia-ctk |
| 4 | nvidia-ctk runtime configure --runtime=docker |
Rewrites /etc/docker/daemon.json so Docker knows about the nvidia runtime |
| 5 | Restart the Docker daemon | Applies the new daemon.json configuration |
A common point of confusion: students often assume they need the full CUDA Toolkit installed on the host. You don't. The host only needs the driver. CUDA itself gets shipped inside the image (or the NGC container), which is exactly what makes containers portable across hosts with different CUDA versions, as long as the driver is new enough.
Once the toolkit is configured, you grant a running container access to GPUs with the --gpus flag:
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
docker run --rm --gpus device=0 nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi--gpus all exposes every GPU on the host to the container; --gpus device=0 restricts it to a specific GPU, which matters on multi-GPU hosts where you want to schedule different containers onto different cards. The environment variable NVIDIA_DRIVER_CAPABILITIES further controls which driver libraries get mounted into the container — compute for CUDA workloads, utility for tools like nvidia-smi, and there are others (graphics, video, display) for more specialized cases.
NVIDIA publishes tagged images on Docker Hub under nvidia/cuda, such as nvidia/cuda:12.6.0-base-ubuntu22.04, with -base, -runtime, and -devel variants offering progressively more of the CUDA toolchain baked in. For real AI training and inference work, NVIDIA's NGC catalog offers fully pre-built containers — for example nvcr.io/nvidia/pytorch:26.01-py3 — that already include matched versions of CUDA, cuDNN, and the framework itself, saving you from dependency hell.
The standard smoke test is:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smiIf configured correctly, you'll see the familiar nvidia-smi table listing your GPU(s), driver version, and CUDA version — proof that the container reached through to the host's hardware.
"If I build my image with GPU support, the image is GPU-accelerated." Not quite. GPU acceleration isn't a property of the image at all — it's a property of how the container is run. The exact same image can run with --gpus all on a GPU-equipped host, or with no --gpus flag on a plain CPU host (falling back to CPU code paths, if the application supports them, or simply failing to find a GPU). The image just needs the right CUDA libraries available; whether a real GPU is attached is decided entirely at docker run time by the toolkit and runtime. "I should be able to do this on my Mac or native Windows." Also not quite. The NVIDIA Container Toolkit requires Linux, because it works by mounting host device files and driver libraries directly into a Linux container's namespace. macOS has no supported NVIDIA GPU passthrough path at all. Native Windows containers can't do it either — but Windows machines can work through WSL2, because WSL2 runs an actual Linux kernel with NVIDIA's WSL driver support, giving Docker something Linux-shaped to pass the GPU into. If your Windows containers are running in Hyper-V isolation instead of WSL2, GPU passthrough still won't work.