🛠️ Lab 12 — From Checkpoint to GGUF: Convert, Quantize, and Compare (50 pts)

Goal: Convert a Hugging Face model checkpoint into GGUF format, produce multiple quantization levels, and compare the resulting file sizes and output quality — first with a zero-setup hosted tool, then by manually running the llama.cpp conversion and quantization workflow yourself. Steps:

1. No-setup pass with GGUF My Repo. Go to the free Hugging Face Space ggml-org/gguf-my-repo (it syncs from the llama.cpp main branch every 6 hours, so it stays current). Point it at a small Hugging Face model checkpoint you fine-tuned in an earlier week (or a small public checkpoint if you don't have your own handy — pick something under \~2B parameters so the job finishes quickly). Use the Space to produce three GGUF quantization levels: Q4_K_M, Q5_K_M, and Q8_0.
2. Record file sizes. Note the resulting file size for each of the three quantization levels produced in Step 1. Build a small table comparing them (level, size in MB/GB, approximate % reduction versus the largest one).
3. Manual walkthrough in Colab. Open a free Google Colab notebook and manually reproduce the pipeline yourself:
• Clone the llama.cpp repository (git clone https://github.com/ggml-org/llama.cpp) and build it (or install the Python requirements needed for conversion).
• Convert your Hugging Face model to an F16 GGUF file using convert_hf_to_gguf.py.
• Quantize that F16 GGUF file with llama-quantize to produce two versions: one at Q4_K_M and one at Q8_0. (Remember: always quantize from the F16 source — never from an already-quantized file.)
4. Run inference locally in the notebook. Load each of your two manually-produced GGUF files (Q4_K_M and Q8_0) using llama.cpp's Python bindings or CLI, and run the same short prompt (3–5 sentences of output) through both.
5. Compare quality. Read the two outputs side by side. Note any differences in coherence, factual detail, or repetition. This is a qualitative spot-check, not a formal perplexity measurement — but it should make the size/quality tradeoff tangible.
6. Write up your findings in a short report (1 page or less) covering: the file-size comparison table from Step 2, a brief description of what you did in Colab, your side-by-side output comparison from Step 5, and one sentence on which quantization level you'd recommend for (a) a laptop with 16GB RAM and (b) a Raspberry Pi with 8GB RAM.

Deliverables: Submit your write-up (PDF or text) plus either your Colab notebook link/export or a screenshot set showing the GGUF My Repo outputs, the convert_hf_to_gguf.py and llama-quantize commands you ran, and the two side-by-side inference outputs.