Schema drift and data contracts
Your pipeline was correct the day you wrote it. Then, quietly, the
upstream changed: a field renamed, an int became a string, a new
key appeared, a mobile release started sending amount in cents.
Nothing crashed — the pipeline kept running and produced subtly
wrong data for six weeks. That slow rot is schema drift, and it
is the default fate of every unguarded pipeline.
The contract move
Chapter 14 gave you the answer for LLM outputs: validate at the boundary. A dataset pipeline is the same picture with a different foreign system upstream. A data contract says, in code:
- which fields must exist,
- what type each one is,
- what to do with anything unexpected.
Run the editor: a 12-line hand-rolled contract catches both classic
drift shapes — a type change and a surprise new field — at the door,
on the first bad batch instead of week six. On your own machine
you'd write this as a pydantic model (Record.model_validate(r))
and get chapter 14's structured ValidationErrors for free;
dedicated tools like Pandera and Great Expectations do the same for
whole DataFrames.
What to do when the contract fires
Three honest options, pick per field:
- Reject the batch — right for label-critical fields. Loud failure now beats silent corruption forever.
- Quarantine bad rows — write them to a
rejected/sink with the error attached; the pipeline continues, and the drop report (mission 35) tells you the damage. - Adapt deliberately — if upstream legitimately changed, update the contract in a reviewed commit. The contract file becomes the changelog of your data's shape — the same date-stamped ratchet discipline chapter 30 used for agent rules.
The one non-option: loosening the contract to "whatever arrives" so it stops complaining. That's deleting the smoke alarm.
Where AI specifically gets this wrong
- No contract at all. Generated pipelines parse optimistically; drift becomes NaNs and wrong joins downstream, far from the cause.
- Try/except-pass around the parse. Chapter 09's bare-except sin, pipeline edition: rows vanish without a count.
There's also the fix that's worse than the bug: auto-widening types. "Just cast everything to string" makes the contract always pass and the data always useless.