📖 Lecture — Autoscaling Economics, Spot Instances, and Build-vs-Buy

By now you know how to build LLM apps and RAG systems. This week we zoom out from "does it work" to "does it work at a price and reliability level a business can live with." That is the job of infrastructure economics: matching capacity to demand automatically, choosing the right compute for the right workload, and knowing when to stop building your own inference stack and just pay a vendor. The two-layer autoscaling pattern. Production GPU inference on Kubernetes almost always uses two autoscalers working together, not one. KEDA (Kubernetes Event-Driven Autoscaling) operates at the pod level. It watches custom metrics — GPU utilization, queue depth, request latency — and scales your inference pods up or down, including all the way to zero when there is no traffic. Karpenter operates at the node level. When KEDA asks for more pods than the cluster has room for, Karpenter provisions new GPU nodes (including Spot capacity) to fit them, typically completing a provisioning cycle in about 60 seconds. When load drops, Karpenter removes nodes that are no longer needed. AWS's own guidance recommends setting consolidationPolicy: WhenEmpty with a consolidateAfter of roughly one hour for spiky inference traffic — this stops Karpenter from aggressively tearing down and rebuilding nodes every time traffic dips for a few minutes, which would otherwise reintroduce the cold-start costs we're about to discuss. Why scale-to-zero isn't free. This directly corrects a common misconception: "just autoscale GPU pods to zero when idle — scaling is instant and free." It is neither. A GPU pod scaling from zero has to pull a container image (often several GB), start the runtime, and load model weights into VRAM (again, often many GB, sometimes tens of GB for larger models). That cold start can take anywhere from tens of seconds to several minutes. During that window, incoming requests either queue, time out, or get dropped. If your scale-to-zero policy is aggressive and your traffic is spiky, you can pay for that cold-start latency dozens of times a day — burning real money in wasted GPU-minutes and, worse, real user trust in failed requests. The fix is deliberate: measure your actual cold-start time, minimize it (smaller/quantized checkpoints, pre-pulled images, model caching), and then decide on an explicit over-provisioning buffer — keeping a minimum of one warm replica, or using a longer consolidateAfter, so capacity doesn't lag demand into a bad user experience. Spot instances: cheap, not free of risk. Spot (AWS) or preemptible (GCP) GPU instances typically cost 50–70% less than on-demand — AWS itself advertises savings up to roughly 90% off on-demand pricing in favorable cases. The catch is reclamation: the cloud provider can take the instance back with as little as two minutes' notice. This corrects the second misconception: "Spot GPUs are just cheaper on-demand GPUs — use them for everything." They are not a drop-in replacement. Spot is well-suited to batch jobs, offline evaluation, fine-tuning runs, and other interruptible workloads — especially when paired with checkpointing and retry logic (SpotServe's token-level checkpointing is a good example: it saves generation progress frequently enough that a reclaimed instance loses seconds of work, not an entire generation job). Spot is a poor fit for latency-critical, synchronous, user-facing APIs with tight P99 SLAs, because an interruption there shows up immediately as a failed or delayed user request. Build vs. buy is a break-even problem, not a philosophy. The third misconception is "self-hosting is always cheaper than paying per-token API prices." Whether that's true depends entirely on sustained utilization. Managed APIs win when traffic is bursty, low-volume, or you're early-stage: you pay only for what you use, with no idle GPU cost and no operational burden. Self-hosting wins when utilization is high and steady — commonly cited as above roughly 50% utilization, or on the order of 10M+ tokens/day per GPU — because at that volume you can amortize quantization and batching optimizations across enough traffic to beat per-token pricing. Below that threshold, the idle GPU-hours you're paying for (whether rented or owned) make self-hosting more expensive, not less. The hidden costs that break the spreadsheet. Two categories of cost are easy to miss. First, self-hosting commonly runs 3–5x the raw GPU rental price once you include DevOps maintenance (roughly \$145K/year for the on-call, patching, and scaling work). Second, engineering time to build the self-hosted stack is real money that never appears on a cloud bill: a fully loaded senior ML engineer (\~\$200K/year) spending three months standing up self-hosted inference infrastructure adds roughly \$50,000 of cost that's invisible unless you go looking for it. Tooling risk compounds this: Hugging Face's TGI (Text Generation Inference) entered maintenance mode in December 2025, with Hugging Face itself recommending teams migrate to vLLM or SGLang — a reminder that the tool you build around today may not be the tool you're maintaining in two years. A simple decision framework ties this together:

Use this table as a starting point for the lab, where you'll build the numbers behind it yourself.