Pagination, retries, checkpoints: ingest that survives reality
Chapter 12 taught the single API call. A dataset pipeline makes thousands of them — and at that scale, the network will flake, the rate limiter will bite, and your laptop will sleep at page 4,817 of 9,000. Ingest code that can't resume is ingest code you'll re-run from zero, repeatedly.
What survives a flaky feed
Pagination. APIs return data in pages (offset/limit, page
numbers, or a next cursor in the response). The loop shape is
always: fetch page → process → advance → stop when the API says
empty/no-next. Cursor-style pagination (the API hands you an opaque
next_cursor) is the most robust — use it when offered.
Retries with backoff. Chapter 12's rule applied in bulk: retry 5xx and 429 with exponential backoff, don't retry 4xx, and cap the attempts. At pipeline scale you add one more: log which page failed, so a stubborn page can be skipped and revisited instead of wedging the whole run.
Checkpoints. Run the editor. The loop persists its position
after every page; when the run "crashes" at page 2, the next run
starts from the checkpoint instead of page 1. In real code the
checkpoint is a tiny file or DB row: {"last_page": 4817}. Cheap
insurance, saved afternoon.
Idempotency: the property that makes resume safe
Resuming means some work may run twice (the crash may have landed after fetching but before checkpointing). So every write must be idempotent — safe to repeat. The usual trick: write records keyed by a stable id (upsert), or write page-files named by page number so a re-write just overwrites the same file. If your pipeline appends blindly, every resume duplicates rows — and lesson 36.3's duplicate check catches it after the damage.
Where AI specifically gets this wrong
- The happy-path loop. Cursor's first draft fetches pages in a
while Truewith no retry, no checkpoint, no cap. It works in the demo and dies at 3am on page 6,000. - Retrying everything. A 401 retried five times is still a 401 (chapter 12) — but now it's also five times slower per page.
- Checkpointing in memory. A checkpoint variable that dies with the process is a diary in disappearing ink. Persist it.