📖 Lecture — Fractional GPUs, MIG, Quantization, and KV-Cache Savings

By now you've built LLM apps and RAG systems, but you've probably run each one on a whole dedicated GPU without asking whether that GPU was actually being used. In production, that habit is expensive. A single A100 or H100 sitting at 15-20% utilization while serving one modest inference workload is money left on the table — and at real-world budgets, that adds up fast. This week is about the two big levers for squeezing more useful work out of the GPUs you already have: splitting one GPU into several isolated slices, and shrinking how much memory and compute each request actually needs. MIG vs. time-slicing: two fundamentally different kinds of sharing NVIDIA's Multi-Instance GPU (MIG) technology, available on Ampere-generation and newer data-center GPUs (A100, L40, H100), hardware-partitions one physical GPU into up to seven isolated instances. Each instance gets its own dedicated slice of compute cores, memory, and cache — enforced by the silicon itself, not by software scheduling. That matters enormously for multi-tenant inference: if Tenant A's job leaks memory, spikes, or misbehaves, it physically cannot starve or crash Tenant B's job, because they don't share a memory pool. This is a hard security and reliability boundary, and it's exactly why AWS Bottlerocket added native MIG support for Kubernetes nodes in March 2025 — cloud operators wanted that isolation guarantee available to container schedulers, not just VM-level deployments. Time-slicing is a different mechanism entirely. It lets multiple jobs share one GPU by taking turns on the same compute and memory pool, switching rapidly between them. This gives you flexibility — you don't have to commit a job to a fixed-size slice — but it comes with no memory isolation. If one workload allocates aggressively or its usage spikes, it can starve or even trigger an out-of-memory (OOM) failure in every other job sharing that GPU. This is the classic "noisy neighbor" problem, and it's the single most important distinction to internalize this week:

The takeaway: MIG and time-slicing are not "two ways to do the same thing, one just shares more." They answer different questions. MIG answers "how do I safely put untrusted or security-sensitive tenants on the same physical GPU?" Time-slicing answers "how do I opportunistically fill idle cycles among workloads I already trust to coexist?" Quantization: shrinking memory without adding GPUs — but it isn't free The second lever is quantization: representing model weights and/or the KV cache in a lower-precision numeric format. Moving from FP16/BF16 down to FP8 or INT8 roughly halves memory footprint. On FP8-native hardware (H100, H200), it can also roughly double throughput, because the tensor cores process FP8 operations faster than FP16. Put together, that can cut effective cost per token by around 50% without buying a single additional GPU. FP8 requires newer hardware (compute capability 8.9+, i.e., Ada Lovelace and Hopper generations); INT8 is more portable and runs fine on older GPUs like the A100 or consumer RTX 40-series cards. But quantization is a trade-off, not a free lunch. INT8 KV-cache quantization is typically near-lossless — you can usually adopt it with minimal quality impact. Push further to INT4 or lower, though, and perplexity rises and generation quality visibly degrades: outputs get less coherent, more repetitive, or subtly wrong in ways that are easy to miss in a quick demo but show up under real use. One nuance that surprises people: the key cache is more sensitive to quantization than the value cache, so an aggressive blanket quantization scheme can hurt more than expected if you don't measure separately. The rule for this week (and for your career): always measure output quality at the precision you actually plan to deploy, on data that resembles your real traffic — don't assume "smaller number format" is a strictly better trade. The KV cache is often the real bottleneck — not the weights A common planning mistake is sizing a GPU deployment based on model weight size alone. In autoregressive generation, every request accumulates a KV (key-value) cache that grows with batch size × sequence length. For workloads with many concurrent users or long contexts, this cache can rival or exceed the memory used by the model weights themselves, and it's frequently the actual limit on how many requests you can serve concurrently — not compute, not weight memory, but KV-cache memory. This is precisely the problem vLLM's PagedAttention was built to solve: it manages the KV cache the way an operating system manages virtual memory, allocating it in fixed-size pages rather than one large contiguous block per request. That sharply reduces fragmentation and lets you pack far more concurrent sequences into the same VRAM — a bigger effective batch size directly translates into lower cost per token served. Tools like vLLM's llm-compressor build on this by applying AWQ, GPTQ, or FP8 quantization to weights and, on hardware with compute capability ≥ 8.9, FP8 quantization to the KV cache itself — though that requires calibrating on a representative dataset before flipping on kv_cache_dtype="fp8" in production. Correcting the misconceptions Let's be explicit about three things people get wrong when they first encounter this material:

1. "MIG and time-slicing are just two ways to do the same GPU sharing." No — MIG hardware-isolates memory and compute per instance; time-slicing shares one memory pool with no isolation at all, meaning a noisy neighbor can OOM everyone else. Choose based on whether you need isolation guarantees, not just "more sharing."
2. "Quantization (including KV-cache quantization) is free — you shrink memory with zero quality loss." Not true. INT8 is close to lossless for the KV cache, but INT4 and lower measurably degrade generation quality, and the key cache degrades faster than the value cache. Always benchmark quality at your chosen precision.
3. "The KV cache is negligible, so I can size a GPU by model weights alone." Also false. The KV cache scales with batch size × sequence length and frequently becomes the dominant memory consumer, directly capping how many concurrent requests you can serve — plan capacity around it explicitly, not as an afterthought.

Master these three ideas and you'll be able to reason clearly about GPU sharing decisions and cost trade-offs in real deployments — which is exactly what this week's lab and discussion ask you to practice.