Train/inference skew — step 1 of 7

Train/inference skew: the same-but-not-same bug

The model learned patterns over features as computed by the training pipeline. If serving computes them even slightly differently, the model receives inputs from a distribution it never saw — and degrades silently. No crash, no error, just worse decisions. This is training/serving skew, one of the most characteristic production-ML bugs there is.

Run the autopsy

Both functions "standardize the value." Training derived mean and std from the data (25, ~11.18); serving hardcoded remembered round numbers (20, 10). Same user, different model input. Every prediction is now shifted — chapter 41's slices would show a degradation nobody can explain, because both codepaths look correct.

The classic causes, all shapes of the same sin (two implementations of one definition):

• Recomputed constants (this demo): means, vocabularies, category encodings re-derived or hand-copied at serving time.
• Reimplemented logic: pandas in the notebook, hand-rolled Python in the API, "the same" tokenization in two libraries.
• Time-travel differences: training features built with full-history batch queries; serving features from a live store with different freshness (chapter 36's as-of guarantee, violated asymmetrically).

The defenses

1. One implementation. The transform is a function/module imported by both training and serving — never rewritten. In sklearn terms, preprocessing lives inside the saved Pipeline object so fit statistics travel with the model artifact (also the clean fix for chapter 38's impute-before-split leakage).
2. The contract checker at both doors (last lesson) — it catches gross skew (type/range) for free.
3. Log a sample of serving-time feature vectors and diff their distributions against training data — that comparison is chapter 47's drift monitoring being born.

Where AI specifically gets this wrong

• "I'll just reimplement it in the server." The generated API code re-derives features from scratch — the bug in its purest form. Import the pipeline; don't translate it.
• Hardcoded normalization constants. Copy-pasted from a notebook output cell, rounded, into the server. The demo you just ran. And the test that fools you: exercising the server with training rows only. Skew hides when test inputs came through the training path. Test through the serving path end-to-end (chapter 46's job).