Title: Quiz 12 — Quantization Foundations: Precision, GGUF, and llama.cpp
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 12 — Quantization Foundations: Precision, GGUF, and llama.cpp

Q: GGUF and quantization are the same thing — using GGUF automatically means your model has been quantized.
- True
- False
A: False

Q: What does a GGUF file package together in a single file?
- Only the quantized weights
- Model architecture, tokenizer info, quantization parameters, and quantized weights
- Only the tokenizer and vocabulary
- Training hyperparameters and optimizer state
A: Model architecture, tokenizer info, quantization parameters, and quantized weights

Q: What is the correct two-step llama.cpp workflow for producing a quantized GGUF file from a Hugging Face checkpoint?
Feedback: Sample/expected answer: First convert the original Hugging Face checkpoint to an F16 GGUF file using ****`convert_hf_to_gguf.py`****, then quantize that F16 GGUF file to the target level (e.g., Q4_K_M) using ****`llama-quantize`****.

Q: It's fine to requantize an already-quantized GGUF file (e.g., go from Q4_K_M to Q5_K_M) instead of starting from the original F16/F32 weights.
- True
- False
A: False

Q: Which statement about Q4_K best describes why it's generally preferred over the legacy Q4_0 format?
- Q4_K uses more bits per weight overall, so it's simply less compressed
- Q4_K is a K-quant that allocates precision more intelligently across tensors, generally yielding better quality than Q4_0 at the same nominal bit width
- Q4_K only works on GPUs, not CPUs
- Q4_K and Q4_0 produce identical output quality in all cases
A: Q4_K is a K-quant that allocates precision more intelligently across tensors, generally yielding better quality than Q4_0 at the same nominal bit width

Q: Lower-bit quantization always means faster inference, regardless of hardware.
- True
- False
A: False

Q: How is quantization error typically measured when comparing a quantized model against its full-precision original?
Feedback: Sample/expected answer: Via perplexity (how well the model predicts held-out text, compared to the full-precision baseline) or KL-divergence between the quantized and full-precision output distributions.

Q: An 8B parameter model is roughly 16GB in F16 GGUF form. Approximately what size and size reduction should you expect after quantizing to Q4_K_M?
- \~4.9GB, about a 70% reduction
- \~1GB, about a 95% reduction
- \~14GB, about a 10% reduction
- \~8GB, about a 50% reduction
A: \~4.9GB, about a 70% reduction