Why architecture matters here
The architecture matters because the economics of batch inference are set almost entirely by accelerator utilization, and utilization is easy to lose. A GPU that can process a batch of 256 rows in the time it takes to process one is wasted by a pipeline that feeds it single rows; a worker that finishes a batch and then waits on a slow output write or a slow input read is a rented accelerator sitting idle. Every architectural choice — how the input is partitioned, how batches are built, how many workers run, how results are written — exists to keep the forward-pass hardware busy. The gap between a naive batch job and a saturated one is routinely 3–5x in cost and wall-clock for identical work, which on a billion-row job is the difference between an affordable nightly run and one that misses its window and blows the budget.
The second forcing function is fault tolerance over long runs. A job scoring a billion rows across dozens of workers for several hours is statistically guaranteed to hit trouble: a spot instance is preempted, a worker OOMs on an outlier record, a transient network error drops a write. Without checkpointing, any such failure means restarting from zero — rescoring everything already done, wasting hours of accelerator time. The architecture therefore tracks progress at shard granularity and makes the whole job resumable: on restart it skips the shards already committed and reprocesses only the outstanding ones. This is not a nicety; for jobs long enough to matter, restartability is the difference between a job that eventually completes and one that never does.
The third reason is correctness of the output under those same failures. Because a shard can fail after partially writing its results and then be retried, the output path must be idempotent — reprocessing a shard must produce the same final state, not duplicate rows. This is usually achieved with deterministic output keys (row id → result) and upsert or write-once-per-key semantics, so a retried shard overwrites rather than appends. Without this, a single retry silently corrupts the output with duplicates, and downstream consumers — a search index, a recommendation table — inherit the corruption. Exactly-once results from an at-least-once execution engine is a defining requirement of the design.
The fourth architectural payoff is decoupling and elasticity. Because batch inference is scheduled and bounded, it can scale workers elastically to the size of the job and the deadline: a huge nightly run spins up many workers to finish in its window, then releases them, paying only for the accelerator-hours used. An autoscaler that watches the depth of the shard queue can add workers when a backlog builds and remove them as it drains, matching cost to work. This elasticity — impossible for an always-on online endpoint sized to peak — is a major reason large-scale scoring is done in batch: you rent exactly the accelerators the job needs, exactly when it runs, and give them back.
A fifth consideration is that batch and online inference are genuinely different systems that happen to run the same model, and conflating them is a common architectural mistake. An online endpoint optimizes tail latency: it keeps a model warm, serves one or a few requests at a time, and cares intensely about the 99th-percentile response. A batch job optimizes aggregate throughput: it can tolerate a slow start, build enormous batches that would be unacceptable for latency, and cares only about total wall-clock and cost for the whole corpus. Trying to run a huge nightly scoring job through an online-serving endpoint wastes money on per-request overhead and small batches, while trying to serve interactive traffic from a batch pipeline delivers minutes-long latencies. Recognizing which regime a workload belongs to — bounded dataset on a schedule versus unbounded stream of latency-sensitive requests — is the first architectural decision, and batch inference is the right answer whenever the input is a large, known set and no human is waiting on any single row.
The architecture: every piece explained
Top row: from dataset to fed device. The input dataset is a bounded corpus — a warehouse table, a set of files, a snapshot of a queue. The partitioner shards it into work units sized so that each is independently processable and restartable, and so the shards distribute evenly across workers (skew here becomes stragglers later). The batch builder is the utilization engine: it groups rows within a shard into batches large enough to fill the accelerator's parallelism, padding or bucketing by length where the model needs it, because an under-filled batch wastes the device. The model workers are replicas of the model — one per GPU, or several sharing a large one — that pull shards, build batches, and run forward passes in parallel across the pool.
Middle row: scoring, recording, and durability. The inference step is the forward pass on a batch — the actual model computation, which on a well-built pipeline is where nearly all the time goes. As shards complete, the checkpoint records which ones are done, so the job knows its progress and can resume. Results flow to the output sink — a warehouse table, a file set, an index — the durable destination the rest of the system reads. The retry / resume path handles failure: a shard that errored (a worker died, a batch OOMed) is returned to the queue and reprocessed, and on a full job restart only uncommitted shards run again.
Bottom row: elasticity and exactly-once. The autoscaler watches the depth of the shard queue and adds or removes model workers to hit the job's deadline without over-provisioning — many workers for a big run, few for a small one, none between runs. Idempotent writes make the output correct despite at-least-once execution: results are keyed by row id and written with upsert or write-once semantics, so a retried shard replaces rather than duplicates. The ops strip names the health signals that matter — throughput (rows/sec), device utilization, cost per row, the count of failed or straggling shards, and output freshness against the schedule.
End-to-end flow
Follow a nightly embedding job that must embed fifty million documents into a vector index before morning. The scheduler kicks off the run; the partitioner reads the source table's row count and splits it into ten thousand shards of five thousand documents each, pushing shard descriptors onto a work queue. The autoscaler sees a deep queue and spins up forty GPU workers. Each worker pulls a shard, and its batch builder groups the shard's documents into length-bucketed batches that fill the GPU — tokenizing, padding within a bucket, and handing each batch to the model for a forward pass that emits embeddings.
As a worker finishes a shard, it writes the embeddings to the output sink keyed by document id — an upsert, so if this shard is ever reprocessed the same keys are overwritten rather than duplicated — and then records the shard id in the checkpoint store as committed. It pulls the next shard and repeats. Across forty workers, forty shards are in flight at once, and the queue drains steadily. Device utilization sits high because the batch builder keeps every GPU fed with full batches and the write path is asynchronous enough not to stall the next forward pass. Throughput and cost-per-row track the healthy saturation the design is built for.
Three hours in, a failure hits: several spot workers are preempted mid-shard. The shards they were processing were not yet committed, so they return to the queue; the autoscaler replaces the lost workers, which pick up those shards and reprocess them from the start of the shard. Because the writes are idempotent, the embeddings for any documents the preempted workers had already written are simply overwritten with identical values — no duplicates enter the index. The job does not restart from zero; it loses only the in-flight shards, a few minutes of work, not three hours. This is checkpointing and idempotency doing exactly their job.
Near the end, a straggler appears: one shard happens to contain unusually long documents, so its batches are smaller and its worker lags while the queue is otherwise empty and other workers sit idle. The scheduler, noticing the tail, re-partitions or speculatively re-runs the straggling shard on a free worker so the whole job is not held hostage by one slow unit, and the run finishes within its window. The outputs land in the index, the checkpoint marks the job complete, the autoscaler releases every worker, and the freshness metric confirms the index was ready before morning — the full cycle of partition, batch, score, checkpoint, write, and resume having run to a clean, exactly-once completion.