Precision, recall, and threshold tradeoffs — step 1 of 7

Precision, recall, and the threshold dial

Your model doesn't output decisions; it outputs scores (chapter 38's conditional probabilities). A threshold turns scores into decisions — and moving it trades one mistake for the other.

The two ratios

Built from the matrix's boxes:

• Precision = TP / (TP + FP) — of everything I flagged, how much was real? The purity of your alarms.
• Recall = TP / (TP + FN) — of everything real, how much did I catch? The coverage of your net.

Run the editor. High threshold (0.80): flag only the most confident cases — precision is a perfect 100%, but recall is 50% (you missed half the positives). Low threshold (0.05): flag everything — recall hits 100% while precision collapses to a coin-flip 50% of false alarms. Same model, three different products, chosen by one number.

That's the honest framing: the threshold is a product decision rendered as a float. Support-ticket triage wants recall (missing an angry customer is worse than an extra look). Auto-refunds want precision (a wrong auto-action is expensive). Chapter 21's judge pipelines made the same call with "only count consistent wins."

F1 and the score-without-a-threshold view

• F1 is the harmonic mean of precision and recall — one number that punishes lopsidedness. Useful for comparing models; useless for choosing your operating point (it hides the tradeoff you just watched).
• ROC-AUC / PR-AUC evaluate the score ranking itself across all thresholds — "does the model put positives above negatives?" Good model-comparison numbers; on rare positives, prefer the PR view (ROC can flatter you by feasting on plentiful true negatives).

Where AI specifically gets this wrong

• The invisible 0.5. model.predict() in sklearn applies a 0.5 threshold silently. Generated code ships it unexamined — choose your threshold on purpose, with predict_proba and a report like this editor's.
• Optimizing F1 because it's there. F1 weighs FP and FN equally; your product almost certainly doesn't.
• Comparing models at different thresholds. Model A at 0.3 vs model B at 0.7 is not a comparison. Fix the operating point (or compare AUCs), then talk.