Structured logs and alerts: the nervous system
A deployed model without monitoring is chapter 24's postmortem waiting for a date. Monitoring starts embarrassingly simply: log every prediction as structured data, then write rules over the stream.
Structured beats prose
Run the editor. Each log entry is a dict — event, model version,
score, latency, error — serialized as one JSON line (chapter 10's
JSONL earning rent again). The payoff over print("predicted ok")
prose: logs become queryable. Error rate, latency percentiles,
score distribution per model version — each is a one-line filter,
exactly like the trace-reading you did in chapter 20, because this
is chapter 20 for classical models. Log the inputs (or a sample
of them) too — they're the raw material for the drift lesson next.
What to log per prediction, minimally: timestamp, model version (the ch.46 response field, now earning its keep), input features (sampled if volume is high), score, latency, and error-or-null.
Alerts: rules with owners
An alert is a queryable condition plus a threshold plus someone who acts: error rate above X%, p95 latency above the budget, prediction volume dropping to zero (the silence alarm — a dead upstream looks like "no errors" otherwise), score distribution collapsing to a constant. Two rules from earlier chapters carry over: thresholds are date-stamped ratchet rules (chapter 30) tuned so alerts stay rare-and-real — an alert channel people mute is no alert channel — and every alert should link to the query that fired it, so the 2am responder starts from evidence (chapter 24's trace-first debugging).
Where AI specifically gets this wrong
- Logs that can't answer questions. Generated servers
print(f"Predicted {x}")— unqueryable prose — and log scores without inputs. Ask for JSON logs with named fields (same effort), and log what the model saw, not just what it said: when drift comes (next lesson), the inputs are the evidence. - Alerts without owners or links. A threshold that pages nobody, or pages with no query attached, converts incidents into ambient dread. Rule + owner + query, or it isn't an alert.