Title: Lecture 3 — CPUs, GPUs, TPUs, and the Memory That Runs Them
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 03 — Computational Resources: CPUs, GPUs, TPUs, and VRAM

📖 Lecture — CPUs, GPUs, TPUs, and the Memory That Runs Them

Every AI system, however impressive its output, ultimately runs on hardware doing arithmetic. This week we look at that hardware — no engineering degree required, just a solid mental model of what each chip is good at.

CPUs, GPUs, and TPUs

The CPU (Central Processing Unit) is the "brain" in every laptop, phone, and server: a small number of powerful, flexible cores running at high clock speed, excellent at sequential logic and branching decisions ("if this, then that"). That's why a CPU juggles a browser, music player, and word processor at once without trouble. What it's not great at is repeating the same simple math operation millions of times simultaneously — exactly what neural networks need. The GPU (Graphics Processing Unit) was originally built to render video game graphics — calculating colors for millions of pixels at once. That same skill, doing the same simple operation on many pieces of data in parallel, is precisely what neural networks need when multiplying large matrices together. Instead of a few powerful cores, a GPU has thousands of small, simple cores built for parallel work, making it dramatically faster than a CPU for deep learning's matrix/tensor math: high-end GPUs deliver roughly 80–300 TFLOPS versus only 1–5 TFLOPS for a CPU. The TPU (Tensor Processing Unit) is a custom chip (an ASIC) Google designed to do one thing extremely well: tensor operations for deep learning, using a "systolic array" architecture wired to push data through a grid of multiply-and-add units efficiently. Unlike a GPU, a TPU isn't general-purpose and isn't something you can buy for your desktop; it's typically only accessible through Google Cloud or Colab.

Chip | Design Philosophy | Great For | Where You'd Find One
CPU | Few powerful, flexible cores | Sequential logic, control flow, everyday computing | Every laptop, phone, server
GPU | Thousands of simple parallel cores | Matrix/tensor math, deep learning training & inference | Gaming PCs, workstations, cloud instances
TPU | Custom ASIC, systolic array | Tensor operations at scale, Google-optimized deep learning | Google Cloud, Google Colab only

The real bottleneck: VRAM, not just speed

The fastest chip doesn't always win — what actually stops beginners is running out of VRAM, the GPU's own onboard memory, separate from your computer's regular RAM. A model must fit in VRAM to run at all. Rule of thumb: inference (running a trained model to get outputs) needs roughly 2 GB of VRAM per billion parameters at fp16 precision, so a 7-billion-parameter model needs about 14 GB just to load and run. Training needs far more — often 2–4x the model's size — because it must also store gradients and optimizer state (bookkeeping numbers optimizers like Adam keep per parameter): roughly 12 bytes of VRAM per parameter for training, versus about 2 bytes per parameter for fp16 inference. That's why you can often run a model on modest hardware but need much beefier (or multiple) GPUs to train or fine-tune it.

Interconnect: how chips talk to each other

With more than one GPU, a new bottleneck appears: how fast the GPUs talk to each other and the CPU. PCIe is the standard link between a GPU and the rest of the system (the latest generation offers around 128 GB/s). NVIDIA's NVLink is a proprietary, much faster GPU-to-GPU connection — up to roughly 1.8 TB/s bidirectional — built so multiple GPUs can share data quickly during large-scale training. For a student on a laptop or a single Colab GPU, interconnect is irrelevant; it only matters once organizations train across many GPUs or machines at once, where slow chip-to-chip communication can become the limiting factor even if each chip is fast.

So... do you need a GPU?

Not always. Here's a practical guide:

Your Task | Do You Need a GPU?
Learning Python, classical ML with scikit-learn | No — CPU is fine
Light fine-tuning of a tiny model | Usually not — CPU works, GPU just speeds it up
Simple inference on a small or quantized model | No — CPU is often sufficient
Training a model from scratch | Yes — GPU (or TPU) strongly recommended
Fine-tuning a large model | Yes — GPU needed, often more than one
Low-latency or high-throughput production inference at scale | Yes — GPU (or specialized hardware)

If you don't own a GPU, that's fine — cloud providers like Google Colab, AWS, Google Cloud, Lambda, and RunPod let you rent GPU or TPU time by the hour (Colab even has a free tier), as long as you know which workloads are actually GPU-bound versus CPU-bound so you don't pay for power you don't need.

Two common misconceptions, cleared up

"A GPU makes every program faster." Not true. GPUs only accelerate highly parallel, matrix/tensor-heavy work. Sequential logic, branching, and most everyday programming see no benefit — and can run slower on a GPU because of time spent transferring data between CPU and GPU. A GPU is a specialist tool, not a universal accelerator. "More VRAM or more GPUs always means proportionally faster training." Also not true. Adding GPUs means they must communicate — sharing gradients, syncing weights — and that communication has real overhead (the interconnect bottleneck above), plus diminishing returns from parallelization generally. Doubling your GPUs rarely doubles your speed; the real answer depends on model size, batch size, VRAM headroom, and how well the workload parallelizes. By the end of this lecture, you should be able to explain why your phone doesn't need a GPU for a calculator app but benefits from one for an AI photo filter — and why "just add more GPUs" isn't a magic speed button.