promptdojo_

SQL quality checks before the notebook — step 1 of 7

ch 36 · sql for ml datasets1/7

The quality checks that run before any training run

Chapter 21 taught eval-first for model outputs. The same reflex applies one step earlier: check the dataset before you trust it. Every check is a one-line query, and together they take five minutes that regularly save a week.

Run the editor. Five queries, five verdicts on a tiny table — and this toy already fails three of them.

The checklist, as SQL

  1. Row count vs expectation. COUNT(*). You expected ~10k users and got 130k? A join exploded upstream (last lesson).
  2. Duplicates. COUNT(*) vs COUNT(DISTINCT ...). Exact dupes double-weight those rows in training; near-dupes across your train/test split are leakage — the model gets quizzed on rows it memorized.
  3. Missingness per column. COUNT(col) vs COUNT(*) (aggregates skip NULLs — last lesson's rule, now working for you).
  4. Label balance. For a 0/1 label, AVG(label) is the positive rate. If it's 0.02, accuracy is the wrong metric and chapter 41 explains what to use instead. If it's 0.5 exactly on real-world churn data, be suspicious of the query, not pleased.
  5. Range sanity. MIN/MAX per numeric column. Negative ages, million-dollar coffees, timestamps from 1970 — outliers and unit mistakes show up here first.

Make it a gate, not a vibe

The checks earn their keep when they run every time the dataset is rebuilt — the same ratchet logic as chapter 30. Wrap them in a script that fails loudly when a check is violated, and put it between "query ran" and "training started." A dataset that skipped its checks is a dataset with unknown bugs, which chapter 24's Klarna-style postmortems taught you is how confident wrong systems ship.

Where AI specifically gets this wrong

  • Generating the query and moving on. Cursor writes plausible SQL; it does not know your expected row count or label rate. Those expectations live with you — write them into the checks.
  • Deduping blindly. SELECT DISTINCT * as a "fix" can delete legitimately repeated events (two identical coffee purchases are real). Distinguish "duplicate rows" from "repeated events" with a proper event id before deleting anything.

And the quiet one: checking once, ever. Data changes. A check that ran in March proves nothing about the June rebuild.

open the editor to run it.