Title: Discussion 13 — Speed, Quality, or Throughput: Picking Your Quantization Strategy
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 13 — Advanced Quantization: AWQ, GPTQ, FP8, and KV-Cache Quantization

💬 Discussion 13 — Speed, Quality, or Throughput: Picking Your Quantization Strategy (20 pts)

Imagine this scenario: your model fits comfortably in VRAM unquantized, but inference is too slow for your use case. You have three levers available from this week's material — AWQ (for calibration speed and robustness), GPTQ (for potentially higher accuracy when your calibration data matches production), or KV-cache quantization (for serving throughput under concurrent load). In your initial post, address:

- Which lever would you reach for first given your own project or a hypothetical use case (be specific about the use case), and why?
- What would "too slow" actually mean in your scenario — is it latency per request, tokens/sec, or how many concurrent users you can serve?
- How would you measure whether your choice actually worked? Name at least one concrete metric (e.g., perplexity delta, tokens/sec, GPU memory footprint, time-to-first-token) and how you'd collect it.

Then respond to at least two classmates — ideally at least one who chose a different lever than you did. Push on their reasoning: would their measurement approach actually catch a regression in quality or an OOD distortion, per this week's lecture on GPTQ's calibration-overfitting risk?