📖 Lecture — Getting GPUs Into Containers, and Where Dev Tools Fit

Last week you got comfortable treating inference serving as a real API-backed service rather than a notebook cell. This week we tackle the plumbing question that trips up almost everyone the first time they try it: how does a Docker container, which is deliberately isolated from the host, get access to a physical GPU sitting on that host? And once you can do that, which tools should you actually reach for while you are still developing and evaluating models, versus when you are ready to serve real traffic?

Why containers don't see GPUs by default

A container shares the host's kernel but not its devices, drivers, or filesystem by default. That isolation is exactly what makes containers portable — but it also means a plain docker run has no idea your host has an NVIDIA GPU attached, let alone how to talk to it. Installing the NVIDIA driver on the host is necessary but not sufficient. The container also needs a way to mount the right device nodes and driver libraries at runtime, and Docker's default runtime doesn't know how to do that on its own. That's the job of the NVIDIA Container Toolkit. It installs an nvidia runtime that Docker can call instead of (or alongside) its default runc runtime. To wire it up, you run:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The first command edits /etc/docker/daemon.json to register the nvidia runtime; the second reloads the Docker daemon so the change takes effect. Skip either step and docker run --gpus all ... nvidia-smi will fail — usually with an error about an unknown runtime or no GPUs being visible — even though the host driver is installed correctly and nvidia-smi works fine outside any container.

Requesting GPUs at run time

Once the toolkit is configured, Docker 19.03+ lets you request GPU access with a simple flag:

docker run --rm --gpus all ubuntu nvidia-smi

--gpus all tells Docker to expose every GPU on the host to the container. You can also target specific devices (--gpus '"device=0,1"') or set the older NVIDIA_VISIBLE_DEVICES environment variable to the same effect. A second environment variable, NVIDIA_DRIVER_CAPABILITIES, controls which driver libraries get mounted inside the container — for example compute,utility mounts the CUDA driver library and utility tools like nvidia-smi, while other values add video encode/decode or graphics libraries. Most inference workloads only need compute,utility; adding capabilities you don't need just bloats what's mounted.

GPUs in Docker Compose

Flags are fine for one-off docker run commands, but production and lab setups are almost always multi-container, which means Compose. Compose requests GPUs declaratively through a deploy.resources.reservations.devices block:

services:
  ollama:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

This tells Compose to reserve GPU resources from the nvidia driver for that service, using the same underlying toolkit you configured for docker run. Docker v27+ also supports the Container Device Interface (CDI), a vendor-neutral standard for describing devices to containers. CDI is worth knowing about because it decouples the "how do I expose a device" mechanism from any single vendor's runtime — useful as you start seeing non-NVIDIA accelerators in the wild — but for this course, the nvidia driver + deploy.resources.reservations.devices pattern is what you'll use day to day.

A quick word on the CUDA toolkit vs. the driver

Here's a distinction that saves a lot of confusion: you do not need the CUDA toolkit installed on the host to run GPU containers. CUDA is backward-compatible at the driver level, meaning a sufficiently new host driver can run containers built against an older (or matching) CUDA runtime bundled inside the image. What you need on the host is a driver version new enough to support the CUDA version baked into the container image — nothing more. The CUDA toolkit, compiler, and libraries can live entirely inside the container.

Dev-only tools: Ollama and LM Studio

Now that GPUs can reach your containers, what do you actually run for day-to-day development? Two tools dominate this space, and it's important to be honest about what they are and are not:

Both are genuinely useful, and both are explicitly dev-only — neither is designed to be your production serving layer. They don't give you the request batching, multi-GPU tensor parallelism, or throughput optimizations that engines like vLLM or TGI (which you'll meet in a future week) are built for. Reaching for Ollama or LM Studio in production because "it already works" is a common trap: it works fine for one user hitting it occasionally, and falls over under real concurrent load. The recommended workflow is a hybrid one: use LM Studio's model browser to interactively try out candidate models — swap between quantizations, chat with them, get a feel for quality and speed — and once you've picked a winner, productionize it with Ollama for lightweight API workloads, or graduate to a real serving engine when you need production-grade throughput and concurrency.

Correcting three common misconceptions

1. "If I installed the NVIDIA driver on my host, Docker should just see the GPU." Not true. The driver makes the GPU usable on the host, but Docker's default runtime has no mechanism to pass that through to a container. You need the NVIDIA Container Toolkit configured with nvidia-ctk runtime configure --runtime=docker and a Docker daemon restart, or --gpus all will fail.
2. "I need the full CUDA toolkit installed on my host to run GPU containers." Also not true. You only need a host driver new enough to support the CUDA version inside your container image — CUDA is backward-compatible at the driver level. The toolkit and compiler can stay inside the image.
3. "Docker layer caching means instruction order in a Dockerfile doesn't really matter." It matters a lot. Docker invalidates the cache for every layer after the first changed instruction. If you COPY your frequently-changing source code before running your (rarely-changing) dependency installation step, every code change forces a full dependency reinstall on rebuild. Order your Dockerfile so volatile instructions (like copying source) come last, and stable instructions (like installing packages) come first.

Understanding this pipeline — driver, toolkit, runtime flag or Compose block, then a dev-only tool sitting on top — is the foundation for everything else this course builds toward. Next week we start looking at what happens when you need more than one user hitting your model at once.