Distributions, sampling, and why small numbers lie
A model never sees "the data." It sees a sample — and every number you compute on a sample (a mean, an accuracy, a positive rate) wobbles depending on which sample you drew. Understanding that wobble is most of practical statistics.
Describe a column in two numbers (then look anyway)
- Mean — the center of mass. Sensitive to outliers: one million-dollar order drags it hard (chapter 36's range check exists for this). The median resists outliers; a big mean-median gap is your skew detector.
- Standard deviation — typical distance from the mean; the spread.
Two columns can share both numbers and look nothing alike, so for
anything important: look at the distribution (a histogram on your
machine — df["col"].hist()), not just the summary.
Sampling variance, live
Run the editor. The true rate is exactly 10%, yet the n=10 samples swing from 30% all the way down to 0% while n=1000 lands right near the truth. Nothing is broken; small samples are just noisy. The wobble shrinks like 1/√n: to halve the noise you need 4× the data.
The direct consequence for your work: an eval on 10 examples (chapter 21) or an A/B readout on 50 users can differ by huge margins purely by luck. Before declaring model B better than model A by 2 points, ask: on how many examples? Would a re-draw flip it? (The re-draw is free to simulate — that's exactly what the editor did.)
Biased samples: the worse problem
Noise averages out; bias doesn't. Training on last quarter's US users and deploying to global traffic; labeling only tickets that got escalated; scraping only public repos — each yields a sample that systematically differs from the population, and no amount of n fixes it. The fix is at collection: know who's in the sample, and slice your evals (chapter 41) to see who's being failed.
Where AI specifically gets this wrong
- Confident conclusions from tiny evals. "Accuracy improved from 80% to 84%" on a 25-case set is a coin-flip story told with a straight face. State the n, always.
- Mean-only summaries. Generated EDA prints
.mean()per column and calls it understanding. Demand median and a look at the tails. - Fixed seeds mistaken for robustness.
random.seed(42)makes the demo reproducible (good!) — it does not make the estimate less noisy. Different things.