Batch vs realtime: when does the prediction need to exist?
Before provisioning anything, ask one question: when is the prediction needed? The answer sorts serving into two shapes with very different engineering bills.
The two shapes
Run the editor for the side-by-side. In words:
Batch: a scheduled job (chapter 37's pipeline, with a model stage bolted on) scores everyone on a cadence and writes results to a table. Consumers read the table. Perfect when decisions tolerate staleness: churn-risk lists for tomorrow's outreach, weekly demand forecasts, nightly document classification. It's dramatically simpler — no always-on service, failures rerun quietly, and throughput is cheap because you control the schedule.
Realtime: lesson 01's API, computing predictions on demand, because the answer depends on this moment — fraud checks inside a checkout, next-word suggestions, dynamic pricing at page load. You pay for that freshness with everything on the right column: latency budgets, uptime, autoscaling, and user-visible failures.
The hybrid that shows up everywhere real: precompute in batch, look up in realtime — nightly scores land in a fast key-value store, and the "realtime" endpoint is a cheap read. You get millisecond responses with batch-grade simplicity, at the cost of freshness. Most "we need realtime ML" requirements are actually this. (LLM APIs mirror the same split, chapter 23: batch endpoints at a discount when nobody's waiting.)
Making the call
Work the question chain: Who consumes the prediction, and when? Would a day-old answer change the decision? What breaks if the service is down five minutes? If staleness is tolerable and the population is enumerable — batch (or the hybrid). Reach for pure realtime only when the input genuinely doesn't exist until request time.
Where AI specifically gets this wrong
- Defaulting to an API. Ask Cursor to "deploy the model" and you get FastAPI scaffolding whether or not anything needed realtime. The nightly-job answer is never generated unless you ask.
- Batch jobs with no failure story. A cron that silently skipped Tuesday means decisions ran on Monday's scores — chapter 37's checkpoint-and-alert habits apply to scoring jobs too. The third failure hides in plain sight: ignoring the hybrid. Teams build (and pay for) live inference to serve values that change once a day.