promptdojo_

Every prediction gets a name — step 1 of 7

If it doesn't have an ID, it didn't happen

Tuesday, 2:07am, your churn model flags customer 88231 and an automated email quietly cancels her loyalty discount. Thursday she files a ticket: "your system decided I was leaving. I wasn't. Why?"

Now answer her. Which prediction was it? The model has served two hundred thousand since Tuesday, and you shipped a new version Wednesday. If all you kept was the final decision, the honest answer is we have no idea — and "we have no idea" is not a sentence you get to say to a customer, a boss, or a regulator.

The last two lessons read agent traces — what the loop did. This one is the layer under it: the inference log, one row per model call, keyed by an ID that exists nowhere else. Run the editor. That's the whole idea in six fields.

The row, field by field

  • prediction_id — minted the instant the prediction is made, and returned to the caller. It rides along in the email, the UI, the support ticket. This is the join key. Without it you're matching complaints to predictions by vibes and timestamps.
  • ts — what the server clock said. Useful for "around Tuesday"; useless as a join key, because Tuesday had 40,000 rows.
  • model_version — which weights answered. You deployed v14 on Wednesday; her prediction came from v13. If you don't log this, every bug report is about a model that may no longer exist.
  • input_hash — a fingerprint of the exact input, not the input itself. Raw inputs are big and full of PII; a hash is eight cheap characters that let you prove "same input in, same answer out" when someone claims the model is flaky. (If you must keep raw inputs, put them in a separate locked-down store — keyed by the same ID.)
  • latency_ms — 41ms is a row. 4,100ms is a story.
  • output — what the model actually said, score included. The email said "discount cancelled"; the log says 0.91. Those are different facts, and only one of them came from the model.

Why unlogged predictions are undebuggable

A prediction with no log row can't be reproduced (you don't have the input), can't be attributed (you don't know the version), and can't even be confirmed to have happened. The complaint conversation collapses to we believe you or we don't — customer support by coin flip. The log row costs microseconds at inference time. Its absence costs you the whole investigation.

Six more steps: you'll pick a log schema that survives a complaint, mint IDs from a counter, join a ticket back to its exact row, and fix a lookup that grabs the wrong prediction with total confidence.