Title: Lab 2 — GPU Passthrough and a Local Ollama + Open WebUI Stack
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 02 — Docker, GPU Runtime, and Dev-Only Tools (Ollama, LM Studio)

🛠️ Lab 2 — GPU Passthrough and a Local Ollama + Open WebUI Stack (50 pts)

Goal: Configure Docker to access your host GPU, stand up a containerized Ollama + Open WebUI stack, pull a small free model, and get a first hands-on look at the CPU-vs-GPU throughput difference. Steps:

1. On a Linux host (or WSL2 on Windows with an NVIDIA GPU), confirm the host driver is working by running nvidia-smi directly on the host. You should see your GPU listed.
2. Install the NVIDIA Container Toolkit following the official install guide. Then configure the Docker runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

1. Verify GPU passthrough:

docker run --rm --gpus all ubuntu nvidia-smi

If this fails with a runtime error, that's the classic missing-runtime issue — re-run step 2, confirm /etc/docker/daemon.json now lists the nvidia runtime, and restart Docker again before retrying.

1. Create a project folder and write a docker-compose.yml that runs two services: ollama (image ollama/ollama, with a deploy.resources.reservations.devices block requesting driver: nvidia, capabilities: [gpu]) and open-webui (image ghcr.io/open-webui/open-webui:main) connected to the Ollama service. Bring the stack up with docker compose up -d.
2. Pull a small free model into Ollama:

docker exec -it <ollama-container-name> ollama pull llama3.2:3b

1. Open Open WebUI in your browser, select the llama3.2:3b model, and confirm you can chat with it end to end (browser → Open WebUI → Ollama → GPU).
2. Benchmark throughput: run the same prompt against llama3.2:3b once with the GPU passthrough active, and once forcing CPU-only (e.g., by temporarily removing the deploy GPU block and restarting the container, or using CUDA_VISIBLE_DEVICES="" if supported by your setup). Record tokens/sec (or generation time) for a fixed prompt in both cases, using ollama run llama3.2:3b --verbose or the API's timing fields.
3. Write a short (150–250 word) reflection comparing the CPU-only and GPU-enabled generation speeds, and note any errors you hit configuring the runtime and how you resolved them.

Deliverables: Submit your docker-compose.yml file, a screenshot of a successful docker run --rm --gpus all ubuntu nvidia-smi output, your CPU-vs-GPU benchmark numbers, and your written reflection (via file upload or text entry).