📖 Lecture — QLoRA: Fine-Tuning Big Models on Small GPUs

Last week you fine-tuned a model with LoRA: freezing the base weights and training small low-rank adapter matrices instead of the whole network. That already cut your trainable parameter count dramatically. This week we add one more trick that cuts memory even further: QLoRA. The core idea, from the original QLoRA paper (Dettmers et al., 2023), is deceptively simple. Keep the frozen base model compressed to 4-bit precision, and backpropagate gradients through that frozen 4-bit model into full-precision LoRA adapters sitting on top. The base model itself never gets updated and never leaves 4-bit storage — only the small adapters are trained and stored at higher precision. The paper's headline result still holds up as the benchmark for this technique: QLoRA made it possible to fine-tune a 65-billion-parameter model on a single 48GB GPU, reaching performance that matched full 16-bit fine-tuning of the same model. That's the promise of QLoRA in one sentence — full fine-tuning quality, a fraction of the memory. Why 4-bit, and why NF4 specifically. A natural worry here is: doesn't cramming weights into 4 bits per parameter destroy a ton of information? It would, if you used a naive quantization scheme. Standard integer quantization (INT4) divides the range of values into equally spaced buckets, which works fine if the underlying values are uniformly distributed. But neural network weights are not uniform — after training, they overwhelmingly cluster in a bell curve around zero, i.e., they're approximately normally distributed. QLoRA's answer is 4-bit NormalFloat (NF4), a quantization scheme whose bucket boundaries are placed to match a normal distribution rather than a uniform one. Because the buckets are denser where the weight values actually live, NF4 is information-theoretically optimal for this kind of data, and it retains more usable signal per bit than plain INT4 at the same bit width. This is the single biggest thing to internalize this week: NF4 is not just "INT4 with a fancier name" — it's a distribution-aware encoding, and that's precisely why QLoRA can compress this aggressively without the quality collapse you'd expect from naive 4-bit rounding. The misconception to retire this week. It's tempting to assume that because the base model is stored in 4 bits, training itself must run faster — smaller numbers, faster math, right? This is not how it works, and it's worth being explicit about it. Quantization to 4-bit primarily shrinks the memory footprint of the frozen base model's weights. The actual math of the forward and backward pass — the matrix multiplications, the gradient computations — still happens in 16-bit (bfloat16 or float16). The 4-bit weights get dequantized on the fly for computation. Activations and optimizer states (for the trainable LoRA adapters) are also kept in 16-bit. So QLoRA's win is squarely about fitting a bigger model's frozen weights into less VRAM, not about making the arithmetic itself cheaper. If anything, the dequantize-then-compute step adds a small amount of overhead per layer — you're trading a little compute overhead for a large memory savings, and for most practitioners that's an excellent trade because VRAM, not compute, is what stops them from touching bigger models at all. Matching hardware to model size in 2026. Practitioner consensus has settled into some reliable rules of thumb you can use when planning a project:

The headline takeaway: you no longer need a data-center GPU to meaningfully fine-tune large models. A single consumer card that a hobbyist can buy is now a legitimate fine-tuning workstation for 7B–13B models, and even a 70B run is within reach on a high-end consumer card like the RTX 4090. Where Unsloth fits in. Unsloth is an open-source library that patches the underlying transformer math and hand-writes custom GPU kernels for the operations PEFT/QLoRA training relies on. It doesn't change the QLoRA algorithm — it makes the same algorithm run more efficiently. In practice this delivers roughly 2.5x faster training and 30%+ less VRAM usage compared to a stock Hugging Face + PEFT + bitsandbytes QLoRA setup, with no loss in final model quality. That efficiency gain is large enough to change what's possible on small hardware: Unsloth can train compact models like Qwen3-4B on as little as 3GB of VRAM, and it's the reason free-tier Colab GPUs (like the T4 you'll use in Lab 7) are viable for real fine-tuning work rather than just toy demos. Under the hood, Unsloth is doing exactly the kind of optimization work you're becoming familiar with in this course: it isn't inventing new math, it's implementing the existing math more carefully so it fits the hardware constraints practitioners actually have. Put together, QLoRA and Unsloth represent the current default recipe for fine-tuning on consumer and prosumer hardware: quantize the frozen base to NF4 to fit it in memory, train small 16-bit LoRA adapters on top to actually learn the task, and let Unsloth's optimized kernels squeeze out the training speed and additional memory headroom. That's what you'll be doing hands-on in the lab this week.