🛠️ Lab 13 — GPTQ vs. AWQ Shootout with FP8 KV-Cache (50 pts)

Goal: Quantize the same small open model with both GPTQ (via GPTQModel) and AWQ (via AutoAWQ), compare their perplexity and generation quality, and then explore FP8 KV-cache quantization in vLLM to observe memory savings for longer context windows — all using free, local, or free-tier tools. Steps:

  1. Open a free Google Colab notebook (use a GPU runtime — T4 is fine). Confirm your GPU is visible with !nvidia-smi.
  2. Install the required libraries: pip install gptqmodel autoawq transformers accelerate datasets. Do not install auto-gptq — it is deprecated and unsupported in current Transformers; use gptqmodel instead.
  3. Load facebook/opt-350m (small enough to fit comfortably in the free tier) along with its tokenizer, and load a small calibration dataset (a few hundred samples of wikitext-2 or similar is sufficient).
  4. Quantize the model to 4-bit using GPTQModel, following the core steps of a standard 4-bit GPTQ walkthrough: build the calibration dataloader, run the GPTQModel quantization call, and save the resulting quantized model.
  5. Separately, quantize a fresh copy of the same base model to 4-bit using AutoAWQ, following its standard calibration and quantization workflow.
  6. Evaluate both quantized models plus the original FP16/BF16 baseline on a held-out text sample: compute perplexity for all three, and generate text from 3-5 shared prompts for a qualitative side-by-side comparison.
  7. Record wall-clock time for each quantization run (GPTQ calibration vs. AWQ calibration) to observe the speed difference described in lecture.
  8. Build a comparison table: model variant, quantization method, perplexity, approximate quantization time, and your qualitative notes on generation quality.
  9. Extension (as GPU access allows): Install vLLM in the same or a fresh Colab session, load a small model, and enable FP8 KV-cache quantization via vLLM's configuration (e.g., the kv_cache_dtype option). Run the same prompt set at a couple of context lengths and note any memory usage differences reported by vLLM versus the default BF16 KV cache. If your free-tier GPU does not support FP8 (FP8 KV-cache benefits most from Hopper/Blackwell hardware), describe in writing what you would expect to measure and why, based on this week's lecture.
  10. Write a short reflection (150-250 words) comparing what you observed against the lecture's claims: did GPTQ calibrate slower than AWQ in your run? Did one method preserve generation quality better? What surprised you?

Deliverables: Submit your Colab notebook (.ipynb, shared link or file upload) containing all quantization code and outputs, your comparison table, and your written reflection, either embedded in the notebook or as a separate text submission.