The confusion matrix: four boxes, no hiding
Accuracy is one number, and chapter 38 showed how it lies on imbalanced data. The confusion matrix is the un-liable version: every prediction lands in one of four boxes, and the boxes have names you'll use for the rest of your career.
The four boxes
Run the editor:
- TP (true positive) — said 1, was 1. The catch.
- FP (false positive) — said 1, was 0. The false alarm.
- FN (false negative) — said 0, was 1. The miss.
- TN (true negative) — said 0, was 0. Correctly ignored.
Accuracy is (TP+TN)/all — it adds the two kinds of right and therefore blends the two kinds of wrong. The matrix keeps them apart, and that separation is the entire game, because the two mistakes almost never cost the same:
- Fraud: an FN is stolen money; an FP is an annoyed cardholder.
- Spam: an FP buries a real email; an FN shows you one spam.
- Medical screening: an FN is a missed disease; an FP is a follow-up test.
Which mistake is expensive is a product decision — the same "what does recovery cost" reasoning the model-picker chapter used — and no single-number metric can make it for you.
Reading a matrix in the wild
sklearn: confusion_matrix(y_true, y_pred) (rows = actual,
columns = predicted — check the docs' convention every time; people
transpose it constantly, and a transposed reading swaps your misses
and false alarms). First questions to ask of any matrix: which
off-diagonal box is bigger, and is the positive row mostly FN?
A model that "scores 88%" while its actual-1 row is mostly misses
is the majority-class baseline wearing makeup.
Where AI specifically gets this wrong
Start with the default failure: accuracy reported alone. That is what generated eval code prints unless you demand the matrix next to it.
- Transposing the matrix. Row/column conventions differ across libraries; generated plotting code mixes them. Label your axes explicitly ("actual" / "predicted").
- Treating FP and FN as interchangeable "errors." The fix for too many FPs (raise the threshold) creates FNs — next lesson makes that dial explicit.