Probability and the baseline you must beat
Before any model, two numbers: the base rate of your label, and the score of the dumbest possible predictor. Skipping them is how teams celebrate models that do literally nothing.
Base rates first
A probability, for our purposes, is a long-run frequency: "2% of
transactions are fraud" means the label's base rate is 0.02. You
computed it in chapter 36 with one query — AVG(label). Every
judgment downstream (which metric, which model, is 92% good?) hangs
on that number.
Run the editor. With an 8% positive rate, the model "always predict 0" scores 92% accuracy while catching zero positives. That's the majority-class baseline — and it's the number your fancy model has to beat before anyone applauds. Chapter 41 builds the metrics that can't be fooled this way; this lesson's job is the reflex: compute the baseline first.
For regression the same reflex is "always predict the mean" (or median) — your model must beat that error, or it has learned nothing.
Conditional probability, the working version
Most useful model outputs are conditional probabilities: P(churn | this user's features) — "probability of churn given what we know." Two practical implications:
- A predicted 0.7 doesn't mean "will churn"; it means "in the long run, about 7 in 10 users who look like this churn." Acting on it (who to call first) is a threshold decision — chapter 41's territory.
- Flipping conditionals is the classic fallacy: P(alert | fraud) being high does not make P(fraud | alert) high when fraud is rare. With a 2% base rate, even a good detector's alerts are mostly false alarms. Base rates again — they never stop mattering.
Where AI specifically gets this wrong
- No baseline in the notebook — and 92% read as good. Cursor jumps straight to a model, and without the majority-class (or predict-the-mean) baseline printed side by side in the same script, an imbalanced dataset's floor gets applauded as an achievement (the editor just proved it).
- Treating scores as certainties. A 0.51 and a 0.99 both round to "positive," and generated code often treats them identically. Keep the probability around; downstream decisions need it.