📖 Lecture — From Fine-Tuning to Serving: Multi-Adapter Inference, Quantization Limits, and the Merge Decision

Congratulations on making it through the fine-tuning and quantization core of AIINFRA 201. This week we shift our attention to the boundary between what you've built and what comes next: serving. AIINFRA 200 (production inference and serving infrastructure) picks up where we leave off, so your job this week is to understand the serving landscape well enough to hand off a working artifact — and a clear recommendation — to that next stage. Multi-adapter serving: one base model, many "fine-tuned" endpoints. Both vLLM and Hugging Face's Text Generation Inference (TGI) support first-class multi-LoRA serving. Recall from earlier weeks that a LoRA adapter is just a small set of low-rank A and B matrices inserted at specific layers — it never touches most of the base model's weights. That property is exactly what makes multi-adapter serving cheap: a serving engine loads the (often large) base model into GPU memory once, and then loads many small LoRA adapters alongside it. When a request comes in specifying which adapter to use, the engine swaps in that adapter's A/B matrices for the duration of the request. The result is that one GPU, running one base model, can effectively serve dozens of "different" fine-tuned models — a customer-support adapter, a summarization adapter, a code-review adapter — without duplicating the multi-gigabyte base weights for each one. This is a massive cost and operational win compared to deploying a fully separate model per use case. The hard constraint: same base, same dtype. This flexibility comes with a firm rule. Every adapter served alongside the base model must have been trained against that exact same base model architecture and the same numeric precision (dtype). An adapter trained against Llama-3-8B cannot be swapped onto a Mistral-7B base, and an adapter trained in bfloat16 against an fp16 base may behave unpredictably. If your team has adapters trained on different base models, you need separate serving instances — one per base model — each hosting its own family of compatible adapters. This is a planning constraint as much as a technical one: know which base model each adapter belongs to before you design your serving topology. Quantization narrows which adapters can be served — silently. When the base model itself is quantized (say, to 4-bit for memory savings), vLLM's LoRA support only extends to adapters that modify attention and MLP projection layers — the standard LoRA target modules you've been using all along. If an adapter also touches layers like embed_tokens or lm_head (which is unusual, but does happen with certain training configurations), or if the adapter is loaded with an incorrect file format or path, vLLM will not throw a clear, loud error. It will simply fail to apply the adapter — and your requests will quietly be served by the unadapted base model. This is one of the most important operational lessons of this week: a running server is not proof that your adapter is working. You must validate adapter behavior with real prompts and expected outputs after deployment, every time. Merging: a real option, with real costs. A LoRA adapter can be "merged" into the base model — mathematically folding the A/B matrices into the original weight matrices so you get back a single, ordinary set of weights. Merging removes the runtime dependency on PEFT-specific loading code and can yield a modest latency improvement, since the serving engine no longer has to apply an adapter at inference time. But merging trades away hot-swapping: once merged, that model is exactly one fine-tuned variant, and you lose the ability to serve many adapters cheaply from one base. Merging a 4-bit quantized model's adapter also introduces real numerical risk: the low-rank approximation in the adapter can interact with quantization-induced rounding in ways that cause measurable performance degradation, distinct from the (already lossy) quantization step itself. This is not "safe and lossless" — it is a deliberate, evaluated engineering tradeoff. The table below summarizes when to choose each serving path:

Validate cheaply before you validate expensively. A practical rule that will save you time and compute budget for the rest of your career: always confirm the behavior you want is achievable with LoRA or QLoRA first. These are cheap and fast to iterate on. If the desired behavior — a tone, a task capability, a domain adaptation — cannot be achieved through LoRA/QLoRA, it is very unlikely that full fine-tuning (which is dramatically more expensive in compute, data, and time) will magically succeed where the cheaper method failed. Full fine-tuning is not a fallback for "LoRA wasn't quite good enough" — it is a different, much costlier tool that should only be reached for when you have a specific, evidence-based reason LoRA's low-rank constraint is the actual bottleneck. Correcting two common misconceptions directly:

1. "Any LoRA adapter can be loaded onto a quantized base model in vLLM without restriction." False. Only adapters targeting supported layers (attention/MLP projections) are guaranteed to apply. Adapters touching unsupported weights like embed_tokens or lm_head, or loaded with the wrong format or path, will silently fail to apply — the server keeps running, but your fine-tuning has effectively vanished. Always test with prompts that specifically probe the fine-tuned behavior.
2. "Merging a 4-bit fine-tuned LoRA adapter back into the base model is a safe, lossless simplification." False. Merging can degrade performance because the low-rank approximation interacts with the base weights, and quantization-induced instability plus rounding inconsistencies can compound the effect. Treat merging as an optimization with a real accuracy cost that must be evaluated, not a free simplification step.

This week's lab puts these ideas into practice: you'll serve your own fine-tuned, quantized model through more than one path, observe the tradeoffs directly, and write the handoff document that a production serving team (like the one you'll imagine in AIINFRA 200) would actually need to make a deployment decision.