Title: Lecture 2 — Training vs. Inference, Latency vs. Throughput
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 02 — AI Workloads: Training vs. Inference, Latency vs. Throughput

📖 Lecture — Training vs. Inference, Latency vs. Throughput

Last week you learned what AI infrastructure is made of: chips, servers, networking, and storage. This week we ask a more practical question: what kind of work is that infrastructure actually doing? The answer usually falls into one of two buckets — training or inference — and the bucket you're in changes almost every infrastructure decision you'll make.

Training: the marathon

Training is the process of teaching a model by having it churn through huge amounts of data, adjusting its internal parameters a little bit at a time until it gets good at a task. Think of it like a marathon runner's months of preparation: it doesn't matter if any single training step takes an extra second, what matters is the total time and total "work" completed across the whole run. Training jobs:

- Run for hours, days, or even weeks without stopping.
- Need enormous amounts of GPU memory (VRAM) to hold the model and the data batches being processed.
- Benefit from high-end GPUs like NVIDIA's A100 or H100, often hundreds or thousands of them working together.
- Are optimized for total throughput — how much data can be processed over the entire job — not the speed of any one step.

Inference: the sprint

Inference is what happens after training: the finished model is asked to actually produce an answer — classify an image, answer a question, flag a fraudulent transaction. This is a sprint, not a marathon. A user (or another system) is waiting right now for a response, so the goal flips from "total throughput over days" to "how fast can I answer this one request." Inference workloads:

- Typically run on smaller, cheaper hardware than training, since a single request needs far less computation than training the whole model.
- Are optimized for response time, not total data processed.
- Can run continuously as a live service (a chatbot) or on a schedule (an overnight batch job) — more on that below.

Common misconception: It's tempting to think inference is "just a smaller, cheaper version of training." It isn't. Training and inference optimize for genuinely different goals — total learning progress vs. speed of a single answer — and that difference drives different hardware choices, different software stacks, and different data-center layouts. A setup that's great for training (giant GPU clusters wired together for maximum combined throughput) is often a poor, wasteful choice for serving live user requests, and vice versa.

Latency vs. throughput

Two terms you'll hear constantly in infrastructure conversations:

- Latency = the time it takes to get back the result of one request. If you ask a chatbot a question, latency is how long you wait for the reply. Low latency matters most for real-time, interactive use cases.
- Throughput = how many requests (or tokens, or records) the system can process per unit of time, in total. Throughput matters most when you care about cost-efficiency at scale — e.g., processing a million records overnight.

Common misconception: "We can just maximize both latency and throughput at the same time, for free." In reality there's a real tradeoff. Pushing latency down usually means processing requests in small batches (or one at a time) and keeping extra capacity provisioned and idle, ready to respond instantly — which costs more and typically lowers overall throughput. Pushing throughput up usually means batching many requests together, which means each individual request waits longer. You can improve both somewhat with better engineering, but past a certain point, better latency costs you money or throughput, and there's no getting both for free.

 | Optimizes for | Typical hardware | Typical duration | Example
Training | Total throughput across the whole dataset | Large clusters of H100/A100 GPUs, high-bandwidth interconnect (e.g., InfiniBand) | Hours to weeks | Training a new chatbot model from scratch
Real-time inference | Low latency per request | Smaller/cheaper GPUs or CPUs, autoscaled | Milliseconds to seconds per request, running 24/7 | A live chatbot or fraud-detection API
Batch inference | Throughput / cost-efficiency | Cheaper or spot compute, no autoscaling needed | Scheduled runs (e.g., nightly) | Re-scoring an entire customer database overnight

Why this matters for infrastructure decisions

Because training and inference have different goals, the infrastructure that supports them looks different too. Training clusters need extremely high-bandwidth networking between GPUs (like InfiniBand) so thousands of chips can share data quickly, and they're usually physically isolated from the infrastructure that serves live user traffic. Inference-serving infrastructure, by contrast, prioritizes things training doesn't need as much: autoscaling (so it can handle sudden bursts of user traffic), redundancy (so a single server failure doesn't take down the chatbot), and geographic distribution (so users around the world get fast responses). This is also where batch vs. real-time inference splits further. A nightly job that re-scores a customer database doesn't need to respond instantly — it can run on cheaper, even "spot" (interruptible, discounted) compute, on a fixed schedule, optimized purely for throughput and cost. A live customer-facing chatbot has the opposite needs: unpredictable traffic spikes, a strict low-latency requirement, and infrastructure that's always on and ready. The takeaway: matching infrastructure to workload type isn't a minor detail — it's one of the most common sources of wasted money and poor performance in real organizations. Using a training-optimized setup to serve real-time chatbot requests is slow and needlessly expensive. Using a real-time-serving setup for a big nightly batch job wastes money on idle capacity and redundancy you didn't need. Learning to ask "is this training or inference? Batch or real-time?" before choosing infrastructure is one of the most useful habits you can build this term.