Title: Lecture 11 — Tensors, Autograd, and Your First Neural Network
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 12 — Deep Learning Tooling: PyTorch

📖 Lecture — Tensors, Autograd, and Your First Neural Network

So far in this course you've used Python and basic tools to prepare data and set up your development environment. This week we open the hood on PyTorch, the most widely used deep learning framework, and look at the three ideas that everything else in deep learning is built on: tensors, autograd, and the training loop.

Tensors: the one data structure that does it all

A PyTorch tensor is, at first glance, just a NumPy array: a grid of numbers with a shape, like a list of numbers (1D), a table (2D), or a stack of tables (3D and beyond). If you've ever worked with a NumPy array, a tensor will feel immediately familiar — you can add them, multiply them, reshape them, and slice them the same way. But a tensor has two superpowers a plain NumPy array doesn't:

1. It can live on a GPU. The exact same tensor operations can run on a graphics card instead of a CPU, which is dramatically faster for the kind of repeated matrix math deep learning relies on.
2. It can track gradients. A tensor can remember the operations that were done to it, so PyTorch can later figure out how to adjust it during training.

This matters because in PyTorch, everything is a tensor: your input data, your model's weights, the predictions your model makes, and the gradients used to improve it. Learning to think in tensors is the foundation for everything else this week — and for every deep learning tool you'll touch after this course.

Autograd: automatic differentiation, not automatic training

Training a neural network means repeatedly asking: "if I nudge this weight a little, does my error go up or down, and by how much?" That nudge-and-check math is called a gradient, and computing it by hand for a real network would be miserable. This is where autograd, PyTorch's automatic differentiation engine, comes in. Here's the mechanism:

- When you create a tensor with requires_grad=True, PyTorch starts silently recording every operation performed on it into a computation graph — a record of "what happened to produce this value."
- When you call .backward() on a final result (typically your loss value), PyTorch walks that graph backwards and computes the gradient of the loss with respect to every tensor that required a gradient, automatically applying the chain rule from calculus.

The result lands in each tensor's .grad attribute. Autograd does the calculus for you — it does not, by itself, change any weights. That distinction trips up nearly every beginner, so let's correct it directly.

Misconception, corrected: Many students assume .backward() computes the gradient and applies the update to the weights in one step. It does not. .backward() only fills in the .grad field on each tensor. Actually updating the weights requires a separate call to optimizer.step() — and, just as importantly, clearing old gradients beforehand with optimizer.zero_grad(). Gradients and weight updates are two distinct steps performed by two different pieces of code.

Building a minimal neural network

A small PyTorch neural network is assembled from a few standard ingredients:

Ingredient | What it does | Typical PyTorch tool
Model structure | Defines the layers and how data flows through them | nn.Module subclass, or nn.Sequential for simple stacks
Layer | Transforms data (e.g., a weighted sum) | nn.Linear(in_features, out_features)
Activation | Adds non-linearity so the network can learn more than straight lines | e.g., nn.ReLU()
Loss function | Measures how wrong the model's predictions are | nn.MSELoss (regression), nn.CrossEntropyLoss (classification)
Optimizer | Uses gradients to update the weights | torch.optim.SGD, torch.optim.Adam

You define your model's shape once (how many layers, how big), then let the loss function and optimizer do the repeated work of measuring and correcting mistakes.

The training loop: the pattern you'll type a thousand times

Every PyTorch training loop, no matter how large the model, follows the same five-step pattern each iteration:

1. Forward pass — feed input data through the model to get a prediction.
2. Compute loss — compare the prediction to the true answer using the loss function.
3. optimizer.zero_grad() — clear out gradients left over from the previous step.
4. loss.backward() — run autograd to compute fresh gradients.
5. optimizer.step() — use those gradients to actually update the weights.

Misconception, corrected: A classic, hard-to-spot bug is forgetting optimizer.zero_grad(). Because PyTorch accumulates (adds to) gradients by default rather than replacing them, skipping this step doesn't crash your program — it silently adds each new gradient on top of the old one, quietly corrupting your training. If your loss behaves strangely or refuses to go down, a missing zero_grad() is one of the first things to check.

GPU awareness: PyTorch will not use the GPU unless you ask

Installing PyTorch does not automatically mean your code runs on a GPU. Tensors and models default to the CPU. To use a GPU, you must explicitly:

1. Check whether one is available: torch.cuda.is_available().
2. Choose a device based on that check, commonly written as: device = torch.device('cuda' if torch.cuda.is_available() else 'cpu').
3. Move both your model and your data to that device with .to(device).

Misconception, corrected: It's tempting to assume that simply having PyTorch installed gives you "free" GPU acceleration. In reality, a compatible GPU, driver, and CUDA-enabled PyTorch build all need to be present, and you still must explicitly move every tensor and model to the GPU device yourself — PyTorch never does this automatically on your behalf.

The good news: if you don't have a GPU on your own laptop, free cloud options like Google Colab's free tier let you enable a GPU runtime and run this exact same device-selection code, so you can see firsthand why .to(device) matters — even without owning any special hardware.