Experiment tracker lite — step 1 of 7

The experiment tracker: memory for your modeling

Two weeks into a project, the only honest answers to "which settings produced the good model?" come from a log — human memory demonstrably doesn't version hyperparameters. An experiment tracker is that log: every run's config in, every run's metrics out, append-only.

Run the forty-line version

Each entry: a config (model, hyperparameters, seed — and in real use, the data version and code commit) and metrics (chapter 41's numbers, per slice where it matters). Note what the log already does for chapter 38's discipline: runs 3 and 4 are the same config with different seeds, and the 0.88-vs-0.84 gap is your measured seed noise — the yardstick any "improvement" must exceed. A tracker that records seeds turns "is it better?" into an answerable question.

The rules that make a tracker trustworthy:

• Append-only. Failed runs stay; deleted embarrassments are how teams re-run the same dead end quarterly.
• Logged by the code, not the human. The training script writes the entry (chapter 10's JSONL is the natural format); humans forget, scripts don't.
• One change per run. The tracker makes chapter 43's rule auditable — a diff of two configs should be one line long when you're claiming causality.

The real tools

MLflow (open source) and Weights & Biases are the standard grown-up versions — same core model (runs, configs, metrics) plus UI, artifact storage, and comparison plots. The concepts you just ran are exactly their vocabulary, which is the point: adopt one when the JSONL file gets crowded, and nothing about your habits changes. (Chapter 21's eval-suite practice plugs straight in: the eval score is a tracked metric.)

Where AI specifically gets this wrong

• Prints instead of logs. Generated training scripts print metrics to a terminal that scrolls away. Redirect that into a logged entry — five lines.
• Config half-captured, comparisons fully broken. The script logs lr but not the data version or seed; the "same" config later mysteriously differs — and then two runs with five diffs produce one victory claim. Log the whole config dict, mechanically; the tracker exists precisely to make sloppy comparisons visible and embarrassing.