Title: Lecture 13 — AWQ, GPTQ, FP8, and the New Quantization Toolbox
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 13 — Advanced Quantization: AWQ, GPTQ, FP8, and KV-Cache Quantization

📖 Lecture — AWQ, GPTQ, FP8, and the New Quantization Toolbox

In Week 12 you learned the basics of post-training quantization. This week we go deeper into the two dominant GPU-centric quantization algorithms — AWQ and GPTQ — plus the newer frontier of FP8 quantization for weights, activations, and the KV cache. These are the tools production teams reach for when a model needs to fit on smaller GPUs or serve more concurrent users without a full retrain. AWQ: protect what matters, round the rest Activation-aware Weight Quantization (AWQ) starts from an empirical observation: only about 0.1%–1% of weights in a transformer are "salient" for output quality. Salience isn't about a weight's own magnitude — it's about which weight channels correspond to activations with large magnitude. AWQ's insight is that protecting these salient channels, rather than optimizing every weight, preserves almost all the model's accuracy. Mechanically, AWQ doesn't hand-pick and store a few weights in higher precision (which would need mixed-precision kernels and complicate deployment). Instead, it computes a per-channel scaling factor: it scales up salient weight channels before quantization and scales down the corresponding activations to compensate, so the math stays equivalent but the quantization error on the important channels shrinks. After scaling, AWQ uses simple rounding-to-nearest — no per-weight optimization loop. Because it only runs activations through the model to find salient channels, rather than computing any Hessian information, AWQ calibrates roughly 5-10x faster than GPTQ. GPTQ: optimize your way to a good INT4 lattice GPTQ takes a different approach. Rather than protecting channels ahead of time, it quantizes weights column-by-column and uses second-order information — the Hessian of the layer's reconstruction error — to adjust the remaining, not-yet-quantized weights to compensate for the error just introduced. This "error compensation" means each quantization decision updates the rest of the row so cumulative error stays small, producing a highly optimized placement of weights onto a discrete INT4 lattice. This is powerful — GPTQ often achieves excellent accuracy on its calibration distribution — but it carries a real risk. Because the Hessian-weighted updates are computed from a specific calibration dataset, GPTQ can overfit to that dataset's statistics. If production traffic looks meaningfully different (a new domain, a different language mix, longer documents), you may see out-of-distribution accuracy distortion that won't show up in your validation perplexity numbers.

Dimension | AWQ | GPTQ
Core idea | Protect salient channels via per-channel scaling | Hessian-weighted error compensation per layer
Uses Hessian? | No | Yes
Calibration speed | \~5-10x faster than GPTQ | Slower (Hessian computation)
Quantization step | Simple rounding after scaling | Optimized placement on INT4 lattice
Main risk | Slightly less optimal per-weight fit | Overfitting to calibration set; OOD distortion
Typical use case | Fast turnaround, robust across distributions | Squeezing max accuracy when calibration matches production

FP8: a third path, and hardware-native Both AWQ and GPTQ target INT4 weight-only quantization. FP8 is a different tool: instead of packing weights into integers, it uses an 8-bit floating-point format — E4M3 (more precision, less range) or E5M2 (more range, less precision). FP8 is supported natively on NVIDIA Hopper and Blackwell GPUs, so it can quantize weights, activations, and the KV cache with hardware-accelerated FP8 compute. vLLM's FP8 KV-cache support offers two scaling schemes: per-tensor (one scale factor for the whole cache) and per-attention-head (a separate scale factor per head, for finer-grained accuracy at a small bookkeeping cost). Why the KV cache matters for serving The KV cache grows linearly with sequence length and must be read from memory bandwidth on every decoded token. Quantizing it from BF16 to FP8/INT8 halves its memory footprint and bandwidth pressure during decode — often the single biggest lever for serving long-context requests at scale. The payoff isn't uniform across engines, though: TensorRT-LLM shows clear throughput gains from KV-cache quantization, while vLLM's gains vary and can be modest for prefill-heavy workloads. Always benchmark your own workload shape rather than assume a fixed percentage improvement. Correcting two common misconceptions "AWQ and GPTQ solve quantization the same way." They don't. GPTQ optimizes the quantized weights themselves, layer by layer, using Hessian-based updates — think "try harder on each weight's placement." AWQ instead optimizes a scaling factor that protects high-magnitude activation channels, then falls back to simple rounding. Same goal, very different mechanism — and that difference is why AWQ calibrates so much faster. "You should quantize keys and values symmetrically — e.g., both to Q4." This is often suboptimal. Keys compute attention scores (via dot products with queries), and errors there distort which tokens the model attends to, so keys need more precision. Values are blended according to already-computed attention weights and tolerate more aggressive quantization. Q4 keys combined with Q8 values often beats uniform Q4 or uniform Q8 across both. The 2026 tooling note As of 2026, AutoGPTQ is no longer supported in Hugging Face Transformers. GPTQModel has fully replaced it as the maintained integration point for Transformers, Optimum, and PEFT — use GPTQModel for any new GPTQ work, as this week's lab does.