📖 Lecture — From Bare Metal to Model-Ready: Linux and GPU Server Fundamentals

Welcome to AIINFRA 200. In AIINFRA 101 you got comfortable with Docker, Kubernetes basics, and cloud fundamentals. Before we containerize and orchestrate GPU inference workloads, we need to understand the layer underneath all of it: a properly configured Linux server. Every managed Kubernetes GPU node, every cloud inference endpoint, and every on-prem AI box is, underneath, a Linux machine running a driver, a kernel, and a set of services. This week we build that foundation by hand so that in later weeks — when things are abstracted behind containers and schedulers — you already understand what's happening one layer down. The baseline stack. A production-ready Linux server for AI workloads is built from a small, predictable set of layers: a long-term-support Linux distribution (Ubuntu 22.04 or 24.04 LTS is the industry default for GPU serving), an NVIDIA driver version that is matched to the kernel module currently loaded in memory, fast NVMe storage sized for model weights (multi-gigabyte checkpoints need to load quickly, and slow spinning disks will bottleneck cold starts), and systemd as the service manager that keeps your inference process alive. None of this is exotic — it's the same Ubuntu server administration you may already know, applied specifically to GPU workloads. nvidia-smi: your first diagnostic tool. The very first command you should run on any new GPU box is nvidia-smi. It is the standard CLI shipped with the NVIDIA driver, and it tells you three critical things at a glance: whether the driver is installed and communicating with the kernel module, which GPU(s) are present and their utilization/memory/temperature in real time, and which processes currently hold GPU memory. Think of it as top or htop, but for the GPU. Before you debug a model-serving issue, before you launch a container, before you file a support ticket — run nvidia-smi first. If it returns a clean table of GPU stats, your driver stack is healthy. If it errors, you have a driver problem to fix before anything else will work. systemd: turning a script into a service. In earlier courses you may have run a Python inference script directly in a terminal, or inside a screen/tmux session, and called it done. That approach does not survive a server reboot, a crashed process, or a lost SSH session in production. The professional pattern is to wrap your long-running model-serving process in a systemd unit file. A unit file is a small configuration file (not a script) that tells systemd: what command to run, what user to run it as, what to do if the process dies (Restart=on-failure), and what environment variables it needs. Once installed with systemctl enable --now my-inference.service, your process starts on boot and restarts automatically on failure — no cron jobs, no manual babysitting. The driver/kernel mismatch error. One of the most common — and most confusing to newcomers — errors you'll encounter is:

Failed to initialize NVML: Driver/library version mismatch

This happens when the NVIDIA driver package on disk has been upgraded (via apt upgrade, for example) but the older version of the kernel module is still loaded in memory from before the upgrade. The userspace library (NVML, which nvidia-smi talks to) and the in-kernel module are now different versions and refuse to talk to each other. The fix is almost always a reboot, which forces the kernel to load the new matching module. This isn't a bug you code your way out of — it's an operational fact of life with driver upgrades, and knowing "just reboot" saves hours of confused troubleshooting. Persistence mode: a small optimization, not a silver bullet. nvidia-persistenced keeps the GPU driver initialized in memory between jobs, avoiding the multi-second re-initialization delay that occurs when the driver has to spin up from a fully idle state. This matters for workloads that spawn short-lived GPU processes repeatedly (batch jobs, serverless-style inference). For a single long-running inference server — which is our default pattern this week — the GPU is already initialized and staying "hot," so persistence mode is largely redundant. It's good to know what it does and when to turn it on, but don't assume it's always necessary. Here's a quick reference comparing the tools and concepts introduced this week:

Correcting three common misconceptions.

1. "I'll just log in as root — it's simpler." Running everything as root removes a critical safety boundary between routine work and system-altering commands. One typo in a root shell can take down the whole server. Standard practice is to log in as a regular, unprivileged user and use sudo only for the specific commands that need elevation (installing packages, editing systemd files, restarting services). This habit costs you nothing day-to-day and saves you the one day it matters.
2. "I should chmod +x my systemd unit file so it runs, like a script." This is backwards. A .service file is configuration, not an executable script — systemd parses it as structured text (it uses an INI-like format with [Unit], [Service], and [Install] sections). Marking it executable does nothing useful and signals a misunderstanding of how systemd works. The thing that actually executes is whatever you specify on the ExecStart= line inside the unit file — that binary or script is what needs the executable bit, not the unit file itself.
3. "My systemd service should have access to the same PATH and environment as my terminal." It won't, by default. When you SSH in and run python3 my_server.py, your shell has already loaded your .bashrc or .zshrc, set PATH, HOME, and possibly activated a virtual environment. systemd services do not inherit any of that — they start in a minimal, explicit environment. If your script calls python3 assuming it's on PATH, the service can fail silently (or log a cryptic "command not found") because systemd never saw the PATH you're used to. The fix is to be explicit inside the unit file: use full absolute paths (e.g., /usr/bin/python3 or the full path to your virtualenv's interpreter) and add any needed variables with Environment= lines or an EnvironmentFile= directive.

Taken together, these five ideas — diagnose with nvidia-smi, supervise with systemd, expect and fix driver/kernel mismatches with a reboot, use persistence mode selectively, and build on a matched OS/driver/storage baseline — are the foundation every later week in this course builds on. Next week we start layering containers on top of exactly this stack.