Data drift and concept drift: how models go stale
Nothing broke, nobody deployed, and the model is getting worse. That's drift — the world moving out from under a model frozen at training time — and it comes in two distinct flavors worth keeping straight.
Data drift: the inputs moved
Run the editor. Training-time users averaged 1.2 tickets; this week's average is ~4 — a four-standard-deviation shift the checker flags instantly, while logins sit exactly where they always were. Data drift means the input distribution changed: a marketing push brings a new segment, a UI change alters behavior, an upstream bug reshapes a field (chapter 37's schema drift is the crash-loud cousin; this is the silent one). The model now extrapolates outside the distribution it learned — chapter 38's biased-sample problem, developing live.
Detection is exactly the demo, industrialized: compare serving windows (from lesson 01's input logs) against training statistics (from chapter 45's registry provenance — this is why artifacts record their data), per feature, on a schedule. Fancier distances exist; the mean/sd comparison catches the drift that matters most first.
Concept drift: the answers moved
Nastier: inputs look identical, but the relationship between input and label changed — fraudsters adapted to the fraud model, a price change altered what predicts churn. Input monitoring shows nothing; only outcome monitoring catches it: collect the labels that arrive late (who actually churned) and score recent predictions against them, per slice (chapter 41), on a rolling window. Label delay is the tax here — fraud confirms in days, churn in months — so the eval is always somewhat behind, which is an argument for leading indicators (score-distribution shifts, lesson 01) alongside the lagging truth.
Where AI specifically gets this wrong
- Monitoring the service, not the model. Generated monitoring covers CPU and 500s. Uptime is necessary; drift is the ML failure mode, and it needs the input/outcome checks above. A close cousin: the one-time "drift analysis." A notebook run once at launch proves nothing about month six — drift checks are scheduled jobs (chapter 37 shapes) or they're theater.
- Treating every drift flag as retrain-now. A flagged shift is a lead: sometimes retraining, sometimes an upstream bug fix, sometimes a new segment needing its own eval slice. Next lesson is exactly this decision.