Slice-based error analysis: where the real bugs live
A single metric describes the average experience. Nobody is the average. The step that turns "the model scores 75%" into "the model fails mobile users" is slicing: compute the metric per meaningful segment and read the spread.
Run it
Overall accuracy 67% — but desktop is at 88% while mobile is at 25%. One aggregate hid a segment where the model is worse than a coin flip. Every production postmortem you read in chapter 24 had this shape: Klarna's bot didn't fail on average, it failed on the long tail of ticket types its evals never sliced.
How to slice
By anything with product meaning and enough rows to trust (chapter 38's n-warning applies — a 12-row slice swings wildly):
- Input segments — platform, region, language, customer tier, ticket category.
- Difficulty proxies — input length, rare-vocabulary rate, time of day, data source.
- Score bands — accuracy among the model's most confident predictions should be excellent; if it isn't, calibration is off and downstream thresholding (last lesson) is built on sand.
Then the discipline that makes it analysis instead of dashboarding: read the failures in the worst slice. Ten actual failing examples, by hand, looking for the shared cause — mislabeled data, a missing feature, a preprocessing bug that eats emoji. Chapter 24's four breakage classes are the sorting bins; chapter 21's "expand the eval set with what you find" is the follow-through.
Where AI specifically gets this wrong
- Aggregate-only eval scripts. Generated evaluation code reports one number per model. Ask for a per-slice table with n's, every time.
- Slicing after the fact to flatter. Hunting for some slice where the new model wins is p-hacking with extra steps. Fix the slices before the experiment (they're your product's segments, not the experiment's output).
- Skipping the reading step. A worst-slice table without ten read examples produces guesses. The examples produce fixes.