Gradient descent by hand — step 1 of 7

Gradient descent by hand: the learning-rate pathologies

You've run the loop twice now. This lesson is the debugging lens: what the loss curve looks like when the loop is sick, and why the learning rate is suspect number one every time.

Run the four personalities

The function is f(w) = w² — a parabola with its minimum at 0 — and the editor descends it with four learning rates:

• lr = 0.01 — crawling. Still miles from 0 after 30 steps. Curve: falling, agonizingly slowly.
• lr = 0.1 — converged. Healthy exponential decay.
• lr = 1.0 — oscillating in place: every step overshoots the minimum and lands exactly mirrored on the far side, bouncing between +5 and −5 forever without shrinking.
• lr = 1.1 — diverging: each overshoot is bigger than the last; w explodes into the hundreds. In real training this is the loss going to inf/NaN within a few steps.

Those four shapes are the vocabulary of loss-curve reading. NaN after step 3 → too high (or a broken loss/feature — check both). Flat-but-nonzero forever → possibly too low, possibly a deeper problem (data, wrong loss, dead units). Smooth fall then plateau → normal; the plateau is where schedules (next lesson) and patience matter.

Why the step size is this touchy

The gradient tells you the downhill direction and local steepness — not how far the valley floor is. A fixed lr bets the same step length everywhere. Steep, curvy regions punish big bets (overshoot); flat regions punish small ones (crawl). All the optimizer machinery in the next lesson exists to make that bet adaptive instead of fixed.

Two practical habits fall straight out:

• Sweep in decades. Try lr = 1e-1, 1e-2, 1e-3, 1e-4 on a short run and read the four personalities off the curves. Fine-tuning between decades is dessert, not dinner.
• When training explodes, halve first, theorize second. It's the cheapest experiment you own.

Where AI specifically gets this wrong

• Copy-pasted learning rates. The lr from a tutorial's dataset arrives welded into your very different problem. It's the first knob to question, not the last.
• "Fixing" divergence by clipping alone. Gradient clipping can mask an lr that's simply too high; the curve limps instead of exploding. Clip and check the lr. And a judging error rather than a coding one: final loss alone. Two runs can end at the same number — one converged, one oscillating around it. The curve tells them apart; the scalar doesn't.