Title: Lab 14 — Quantize, Measure, and Partition
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 14 — Fractional GPUs, MIG, Quantization, and KV-Cache Savings

🛠️ Lab 14 — Quantize, Measure, and Partition (50 pts)

Goal: Get hands-on with the two levers from this week's lecture. First, quantize a real model at multiple precisions using free open-source tools and measure the size/VRAM/quality trade-offs yourself. Second, since free-tier GPUs don't expose MIG hardware, complete a concept/design exercise planning a realistic MIG partition layout for a shared A100-80GB using NVIDIA's own official guidance. Steps:

1. Set up a free GPU environment. Open a free-tier Google Colab or Kaggle notebook with GPU runtime enabled (T4 or similar is fine). Confirm GPU access with nvidia-smi.
2. Install llama.cpp. Clone and build the free, open-source llama.cpp project (or install a prebuilt release with the llama-quantize tool). Download a small open-weight model in FP16 GGUF format (a 1B–3B parameter model keeps this feasible on a free GPU/CPU).
3. Quantize to three precisions. Use llama-quantize to convert your FP16 GGUF model into three variants: Q8_0, Q5_K_M, and Q4_K_M. For each variant, record the resulting file size on disk.
4. Run and measure each variant. Load each quantized model with llama.cpp's CLI or server and run the same 3-5 prompts through all three versions. For each, record: peak VRAM/RAM usage (via nvidia-smi or system monitor), rough generation speed, and your qualitative judgment of output quality (coherence, correctness, repetition). Note how this compares directionally to the reference data point from the lecture: Llama-2-70B shrinks from \~140GB at FP16 to \~38GB at Q4_K_M, a 3.7x reduction.
5. Cross-check with bitsandbytes. In the same or a new notebook, load one Hugging Face Transformers model three ways using the free bitsandbytes library: full FP16, 8-bit, and 4-bit (NF4). After each load, log peak VRAM via nvidia-smi. Confirm the memory pattern matches what you saw with the GGUF quantization levels.
6. Design a MIG partition layout (concept lab). Free-tier notebooks don't expose MIG-capable hardware, so this step is a planning exercise using the free, official NVIDIA MIG User Guide. Assume you have one A100-80GB and a mix of small inference jobs. Choose real MIG profiles (from 1g.10gb up to 7g.80gb) to cover your job mix, and produce a table showing: profile chosen per job, number of instances, memory per instance, and your estimated utilization and cost-per-job versus simply running all jobs sequentially on one whole, unpartitioned GPU.
7. Write up your findings. In a short report (1-2 pages), present your quantization measurements table, your MIG partition layout table, and 3-4 sentences reflecting on where you'd draw the line on acceptable quality loss, and when you'd choose MIG partitioning over running whole GPUs.

Deliverables: Submit your notebook(s) or exported script(s) showing the quantization commands and measurements, plus your written report (PDF or doc) containing both required tables (quantization results; MIG partition layout) and your reflection — upload as a file or paste as text.