Loss, forward, backward, step — step 1 of 7

Forward, loss, backward, step: the four-beat loop

Every deep-learning training run — the toy in this editor and the runs that produce frontier models — is the same four beats repeated:

1. Forward — run the model on a batch, get predictions.
2. Loss — one number measuring how wrong they are.
3. Backward — gradients of the loss for every parameter (chapter 42's autograd).
4. Step — nudge every parameter against its gradient.

Run the editor and watch the loop pull w, b from (0, 0) toward the true (3, 1) while the loss collapses. Then read the torch version — the beats are labeled by API calls:

for xb, yb in dataloader:              # batches, not the whole set
    optimizer.zero_grad()              # (accumulation rule, ch.42)
    pred = model(xb)                   # 1. forward
    loss = loss_fn(pred, yb)           # 2. loss
    loss.backward()                    # 3. backward
    optimizer.step()                   # 4. step

Losses: the objective in one number

The loss function is the definition of "wrong," so it must match the task: mean squared error (MSELoss) for regression — you've now implemented it twice — and cross-entropy (CrossEntropyLoss) for classification, which rewards putting probability on the true class. Torch's cross-entropy expects raw scores ("logits") plus integer class labels — feeding it already-softmaxed probabilities is a classic generated-code bug that trains, just worse.

Batches and epochs

Real datasets don't fit in one gradient computation, so the loop processes batches (say 32 rows at a time — one noisy-but-cheap gradient each) and calls one full pass over the data an epoch. Batch gradients wobble around the true downhill direction; that wobble is why the loss curve jitters and partly why training escapes bad flat spots. This editor's toy uses the whole 4-row dataset per step — same loop, batch size "all."

Where AI specifically gets this wrong

• Beat order scrambled. zero_grad after backward, or step before backward — the loop runs and learns garbage. Check the four beats in order in any generated loop, first.
• Wrong loss for the task. MSE on class labels, or cross-entropy on pre-softmaxed outputs. The loss line deserves a minute of reading, not a glance.
• Loss printed once, not curved. A single final loss hides divergence, plateaus, and the moment things broke. Log it every N steps and look at the curve — next lessons depend on it.