By now you've fine-tuned a model and watched it pick up a new skill or a new voice. This week we tackle a different, practical problem: how do you get that model off a data-center GPU and onto a laptop, a mini PC, or even a Raspberry Pi? For most local and edge deployments, the answer runs through two related but distinct ideas — GGUF and quantization. Understanding the difference between them is the single most important thing you'll take from this lecture.
GGUF ("GPT-Generated Unified Format") is a single-file binary format designed by the llama.cpp project. A GGUF file packages everything an inference engine needs: model architecture, tokenizer information (vocabulary, special tokens, merge rules), quantization parameters, and the weights themselves. Because everything lives in one file, deploying a model becomes as simple as copying a .gguf file onto a machine — no separate config files or tokenizer directory to juggle. That simplicity is why GGUF is the standard for running LLMs on CPUs, laptops, and edge devices via llama.cpp and downstream tools like Ollama, LM Studio, and koboldcpp. Here's the misconception worth naming immediately: GGUF and quantization are not the same thing. GGUF is just the container. You can have an F16 (16-bit) GGUF that isn't meaningfully quantized at all, or one quantized down to 4 bits per weight. The quantization level is a property of what's inside the container, not the container itself — so "which quantization level?" is always the real follow-up question, since that's what determines size, speed, and quality.
Quantization reduces the number of bits used to represent each weight, trading some numerical precision for smaller file size and (often, but not always) faster inference. llama.cpp supports a range of quantization levels, and the naming convention tells you a lot if you know how to read it.
| Level | Bits/weight | Block size | Typical use case |
|---|---|---|---|
| F16 | 16 | n/a | Unquantized-ish baseline; highest fidelity, largest file, used as the source for further quantization |
| Q8_0 | 8 | 32 | Near-lossless quality, \~2x smaller than F16; good when you have RAM to spare |
| Q4_0 | 4 | 32 | Basic legacy 4-bit scheme; smallest of the "simple" quant types, but weaker quality per bit |
| Q4_K (e.g., Q4_K_M) | 4 (mixed) | varies by tensor | "K-quant": smarter per-tensor bit allocation, generally better quality than Q4_0 at the same nominal bit width |
The "K-quants" (Q4_K_S, Q4_K_M, Q5_K_M, etc.) are generally the better default over legacy Q4_0/Q5_0/Q5_1 schemes, because they don't apply one crude scale to every block uniformly — they allocate precision more intelligently across tensors, recovering quality a flat 4-bit scheme would lose. The suffix (_S, _M, _L) tracks a small quality-for-size dial within the same bit width. How do we know a quantization level is "good enough"? We measure perplexity (how surprised the model is by held-out text; lower is better, and closer to the full-precision baseline is the goal) or KL-divergence between the quantized model's output distribution and the original's. Both give you a number, not a vibe, so you can make an informed tradeoff instead of guessing. This sets up the second misconception directly: lower-bit quantization does not automatically mean faster inference. Bit width determines memory footprint and bandwidth requirements, but actual speed also depends on whether your hardware and llama.cpp's compute kernels have optimized code paths for that specific format. A Q4_K_M model might run faster than Q8_0 on one machine because it moves less data through a bandwidth-constrained CPU, while on a setup with strong 8-bit kernel support the gap could be much smaller than expected. Never assume "fewer bits = faster" without checking — it depends on your hardware and llama.cpp's kernel support for that quant type.
The standard llama.cpp quantization pipeline has two steps, always run in this order:
convert_hf_to_gguf.py. This preserves full precision — it's a format conversion, not a lossy compression step.llama-quantize tool.The costly beginner mistake — worth burning into memory — is requantizing an already-quantized file instead of starting fresh from the original F16/F32 weights. If you take a Q4_K_M file and try to "convert" it again into Q5_K_M, you are not recovering precision that was already thrown away in the first pass; you're compounding error on top of error. Always keep (or regenerate) the F16 GGUF as your source of truth, and quantize from that every time you want a different level. To make this concrete: an 8B-parameter model in F16 GGUF form is roughly 16GB. Quantized to Q4_K_M, it shrinks to around 4.9GB — about a 70% size reduction — while retaining most of its output quality. That's the difference between "needs a workstation GPU" and "runs comfortably on a laptop or even a Raspberry Pi." This week's lab has you produce and compare exactly these tradeoffs yourself.