Title: Discussion 12 — Quantizing a 70B Model for 8GB of Edge RAM
Course: AIINFRA 201: Model Adaptation — Fine-Tuning & Quantization > Section: Week 12 — Quantization Foundations: Precision, GGUF, and llama.cpp

💬 Discussion 12 — Quantizing a 70B Model for 8GB of Edge RAM (20 pts)

Imagine you need to serve a 70B-parameter model on edge devices that only have 8GB of RAM. Is GGUF quantization your answer here — and if so, which quantization level would you pick, and why? In your initial post, address:

- Is quantization alone enough to fit a 70B model into 8GB, or are there other constraints (RAM overhead, context length, OS/runtime footprint) you'd need to account for as well?
- Would you lean toward a smaller/lower-bit quant (like Q4_0 or Q4_K_M) for footprint, or a larger one (Q8_0) if you had any headroom — and what's your reasoning in terms of the speed-vs-quality tradeoff discussed in the lecture?
- Would you consider an alternative to a 70B model entirely (e.g., a smaller base model) if the RAM math doesn't work out, and why?

Post your initial response, then respond to at least two classmates — ideally ones who reached a different conclusion than you — and engage with their reasoning about the tradeoff.