Two keys that are 95% identical get attention weights of [0.96, 0.04] — practically a hard argmax over near-twins. The bug: raw dot products grow with DIMENSION (16 terms of ~4 each ≈ 64), so tiny relative differences become huge absolute gaps and softmax saturates. That's exactly why attention divides scores by √dim. Add the scaling.
The break is on line 13 — but read the whole snippet first.