Title: Lecture 6 — Teaching a Big Model New Tricks Without Retraining All of It
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 06 — Parameter-Efficient Fine-Tuning: LoRA and PEFT

📖 Lecture — Teaching a Big Model New Tricks Without Retraining All of It

By now you know how to load a pretrained model and run inference on it. But what happens when the base model almost does what you need, just not quite? Maybe it needs to speak in your company's tone, follow a specific output format, or get better at a narrow domain like legal summarization. Full fine-tuning, updating every one of a model's billions of parameters, works, but it is expensive: it needs enormous GPU memory to store gradients and optimizer states for every weight, and it produces a full-size copy of the model for every task. For adult learners running experiments on a laptop or a single free Colab GPU, that is simply not realistic. This week introduces the technique that made model customization accessible to almost everyone: Low-Rank Adaptation (LoRA), and the library that makes it easy to use, PEFT. The core idea of LoRA. LoRA starts from an observation: when you fine-tune a large model on a new task, the update to each weight matrix tends to have a low "intrinsic rank." In other words, you don't need a full-rank update to capture what the model needs to learn. So instead of updating the original weight matrix W directly, LoRA freezes W completely and adds a side path: two small matrices, A and B, whose product (B·A) approximates the update. If W is a large d×d matrix, A and B are only d×r and r×d, where r (the rank) is a small number like 4, 8, or 16. During training, only A and B receive gradients. W never changes. At inference, the model's output is simply W·x + (B·A)·x, so the frozen base knowledge stays intact while the small adapter layers steer the output toward your new task. This matters enormously for trainable-parameter count. A rank-8 adapter on a 7-billion-parameter model might touch well under 1% of the total parameters, yet still recover most of the performance gain you'd get from full fine-tuning. That is the promise PEFT delivers on. PEFT: the library that makes this practical. Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) library gives you a unified, consistent interface for LoRA and related methods. You define a LoraConfig (specifying rank, alpha, target modules, and dropout), wrap your base model with get_peft_model(), and PEFT handles inserting the adapter layers, freezing everything else, and reporting how many parameters are actually trainable. The base model's weights on disk are untouched, which means you can train several different adapters for several different tasks, all against the same frozen base model. Rank and alpha are coupled, not independent. A common misconception is that you can pick rank (r) and alpha (α) freely, as though they were unrelated dials. In reality, nearly every LoRA implementation scales the adapter's contribution by α/r before adding it back to the frozen weight. That means rank and alpha together determine the magnitude of the update, not just its shape. If you double the rank without adjusting alpha, you change the effective scaling and can destabilize training, causing loss spikes or a model that either barely adapts or overcorrects. The widely used guideline is alpha = 2 × r: if r = 8, set alpha = 16; if r = 16, set alpha = 32. This keeps the update magnitude in a well-behaved range as you experiment with different ranks. Where you apply LoRA matters as much as how much. Another common misconception is that adapting only the attention modules (query and value projections) is enough for good results. Early LoRA papers popularized attention-only targeting, and it does work reasonably well. But more recent research shows that targeting the MLP/feed-forward layers, or all linear modules in a transformer block, often outperforms attention-only adaptation. This is because attention and feed-forward layers appear to have different "intrinsic dimensionality" for a given adaptation task; feed-forward layers frequently need more capacity to absorb task-specific knowledge. In practice, if your hardware budget allows it, targeting all-linear modules is a strong default, and if you must be selective, don't assume attention alone is sufficient. Table: choosing your LoRA configuration

Decision | Guideline | Why it matters
Rank (r) | Start at 8; try 4 for tiny adapters, 16+ for harder tasks | Higher rank = more capacity, more trainable params, more memory
Alpha (α) | Set α = 2r (e.g., r=8 → α=16) | Keeps the α/r scaling factor stable as rank changes
Target modules | Prefer MLP/feed-forward or all-linear over attention-only | FFN layers often need more adaptation capacity than attention
Dropout | 0.05–0.1 on adapter layers | Regularizes the small adapter against overfitting
Inference mode | merge_and_unload() for single-task deployment; keep separate for multi-adapter swapping | Merged = zero extra latency; separate = flexible, swappable

Merging vs. keeping adapters separate. Once training is done, you have a choice. Calling merge_and_unload() folds the adapter matrices back into the base weights, producing a single standalone model with no architectural difference from the original, and therefore zero additional inference latency. This is ideal when you're deploying one adapter for one purpose. Alternatively, you can keep the adapter separate from the base model, which lets you load multiple adapters onto the same frozen base and swap between them, or even stack them, without duplicating the (large) base model on disk. This is powerful when you're serving many fine-tuned "personalities" or task specialists from a single deployed base model. By the end of this week's lab, you'll have built LoRA adapters yourself, seen the trainable-parameter savings firsthand, and compared a merged model against an adapter-plus-base setup, so these ideas stop being abstract and become something you've actually measured.