Autograd as a dependency graph — step 1 of 7

Autograd: the tape recorder inside every tensor

Chapter 39 computed a gradient by hand for one parameter. A neural network has millions. Nobody derives those by hand — the framework does, automatically, and the mechanism is simpler than its reputation.

Run the twenty-line version

Every Value in the editor remembers how it was made — its parent values and a little function that knows how to pass gradient backward through its operation. The forward pass computes y = w*x + b and, as a side effect, builds that history graph. Calling y.backward() walks the graph in reverse, depositing into each .grad exactly how much a nudge to that value would move y. The chain rule, as a data structure.

Check the printout: dy/dw is 3 (x's value), dy/dx is 2 (w's value), dy/db is 1. You just backpropagated.

The same thing, in torch

import torch
w = torch.tensor(2.0, requires_grad=True)
x = torch.tensor(3.0)
b = torch.tensor(1.0, requires_grad=True)

y = w * x + b        # forward: torch records the graph
y.backward()         # backward: gradients appear
w.grad               # tensor(3.)

requires_grad=True marks the tensors you're training (the parameters); everything computed from them gets taped; backward() fills .grad. Two operational facts with daily consequences:

• Gradients accumulate. A second backward() adds into .grad rather than replacing it. That's why every training loop zeroes gradients each step (optimizer.zero_grad()) — forget it and your gradients are a running sum of every step so far, which trains, badly, and looks like a mystery.
• Inference doesn't need the tape. Wrapping prediction in with torch.no_grad(): skips graph recording — less memory, faster, no accidental training. Pair it with model.eval(), which does a different job: flipping layers like dropout and batch-norm into their evaluation behavior. (Common mixup: eval() does not touch autograd.)

Where AI specifically gets this wrong

• Missing zero_grad(). The classic generated-loop bug, and now you know its exact mechanism: accumulation is a feature being misused.
• Inference with the tape on. Generated eval code that forgets no_grad() quietly doubles memory use — found only when the batch size mysteriously can't grow.
• Logging loss, the tensor. Appending it to a metrics list keeps its whole computation graph alive — a leak-shaped bug that surfaces as creeping memory use.
• .item() is what the code meant. The float alone, graph released. A one-token fix, once you know which token.