Retraining triggers and human review — step 1 of 7

Retraining triggers: when, and when not, to retrain

Retraining is not free: it costs compute, review attention, and — every deploy — a small chance of shipping something worse. So "when do we retrain?" deserves a written policy, not a mood. The policy is a function over monitoring signals, and it fits in the editor.

The three trigger families

Run it. Week by week, the rule composes three kinds of evidence:

1. Schedule — retrain at least every N days regardless. Insurance against slow drift nobody's dashboard caught, and it keeps the retraining pipeline itself exercised (a retrain path unused for six months is chapter 43's untested restore).
2. Input signals — lesson 02's drift flags. Week 2 drifts on tickets but outcomes still hold: a judgment zone — this policy retrains; a stricter one might watch one more week.
3. Outcome floors — rolling accuracy against late-arriving labels (lesson 02). Week 3 breaks the floor and the decision makes itself.

Whatever fires, the route is always the same: through chapter 46's gated pipeline — retrain, evaluate against the incumbent, canary, promote. A trigger authorizes a candidate, never a deploy. And sometimes the right response to a trigger is not retraining at all: drift caused by an upstream bug wants a data fix; a new segment wants an eval slice and maybe its own model. The trigger starts an investigation; the gate decides shipment.

Write it down, version it

The policy — thresholds, windows, floors — is a config file with an owner and a date (chapter 30's ratchet, once more). When the 90-day schedule or the 0.80 floor changes, that's a reviewed diff, so future-you knows which regime any past decision was made under (the tracker and registry record the rest).

Where AI specifically gets this wrong

• "Retrain nightly" as a default. Generated MLOps enthusiasm. Frequency needs a reason, and "it's what the tutorial did" isn't one.
• Nightly without gates or data. Ungated, it's a nightly regression lottery; with thin new data, it's churn for nothing. Two distinct ways the same cron goes wrong.
• Triggers with no investigation step. Auto-retraining on a drift flag caused by an upstream schema bug bakes the bug into the model (chapter 37's contract should have caught it first).
• Unversioned thresholds. A floor someone once typed is tribal knowledge in a config's clothing. Date, owner, rationale — same as every other rule in this course.