Moving Deferrable Agent Work to Provider Batch APIs: What Can Wait, a Submission Ledger, Partial Results, Expiry and Resubmission

Half price for work nobody is waiting for

Every major model provider now offers an asynchronous batch interface with the same bargain: submit many requests at once, accept a completion window instead of a response, and pay less. The OpenAI Batch API documents a 50 percent discount and a 24-hour completion window. The Claude Message Batches API charges all usage at 50 percent of standard prices and states that most batches finish in less than an hour, with a 24-hour expiry. The Gemini Batch API processes requests at 50 percent of the standard cost with a target turnaround of 24 hours.

Agent systems generate a surprising amount of work that nobody is waiting for: nightly regression evaluations, judge scoring of yesterday's traces, backfills when a prompt or schema changes, memory consolidation, report drafting. Teams often run all of it through the synchronous path because that is where the client code already is. This article covers which work can move, the arithmetic, and the part that takes engineering: keeping exact track of thousands of requests whose results arrive late, out of order, partially, or not at all.

What can wait, and what cannot

The last row deserves care. A ten-step loop cannot be batched as a unit, because step two depends on the output of step one. What can be batched is one step across many independent runs: an evaluation of 5,000 tasks with three model turns each becomes three batches submitted in sequence, each containing one turn of every task. In the worst case that is three completion windows of wall-clock time, which is acceptable for a weekly benchmark and not for a release gate that must finish tonight.

The arithmetic

Take a nightly evaluation of 40,000 cases, each sending 3,000 input tokens and receiving 400 output tokens. With illustrative prices of 2 dollars per million input tokens and 8 dollars per million output tokens, the synchronous run costs 120 million input tokens times 2, which is 240 dollars, plus 16 million output tokens times 8, which is 128 dollars: 368 dollars a night. At half price it is 184 dollars, a saving of about 67,000 dollars a year for one job. Substitute your own prices; the ratio is what the providers document.

Two details change the sum. Prompt caching and batching combine on some providers but with conditions: the Claude batch documentation notes that batches can take longer than the default five-minute cache lifetime and suggests the one-hour cache duration for shared context, and that a request with max_tokens of zero for cache pre-warming is not supported inside a batch. And the billing of unfinished work differs: OpenAI states that tokens consumed by completed requests in an expired batch are charged, while Claude states that errored, canceled and expired requests are not billed.

End-of-window behaviour differs

A batch can run out of time. What you get back then is documented differently by each provider, and a client written for one silently mishandles another.

All three agree on one thing that breaks naive clients: results come back as JSON Lines in an order that need not match the input, so every request carries an identifier you choose (custom_id for OpenAI and Claude, a user-defined key for Gemini input files), and that identifier is the only safe join key. Claude restricts it to 1 to 64 characters of letters, digits, hyphens and underscores; using that pattern everywhere keeps one ledger portable.

A submission ledger

The provider tracks batches; your system must track items. Keep a ledger row per unit of work with its state, the batch it is in, an attempt counter and the last error. The runnable example in agent-decision-lab implements the cycle with a simulated provider for each end-of-window behaviour:

1. Plan. Select pending items under the attempt limit and chunk them under both the request-count and the byte limit.
2. Submit. Increment the attempt first, then derive the custom_id as item id plus attempt, so a resubmitted item can never be confused with its earlier attempt.
3. Reconcile. Join results to submitted items by custom_id. Succeeded results are applied to the downstream store keyed by item id. Expired, canceled and missing requests return to pending. Invalid requests go to review and are never resubmitted.
4. Repeat until nothing is pending, then report review items to a person.

Abridged from examples/python/batch_submission_ledger in agent-decision-lab.

def reconcile(self, batch_id, results, max_attempts=3):
    expected = {self.items[i].custom_id: self.items[i] for i in self.batches[batch_id]
                if self.items[i].batch_id == batch_id and self.items[i].state == "submitted"}
    reported = set()
    for r in results:                       # order is not guaranteed
        item = expected.get(r["custom_id"])
        if item is None:
            continue                        # unknown, or a late result of an older attempt
        reported.add(r["custom_id"])
        if r["type"] == "succeeded":
            self.applied.setdefault(item.item_id, r["text"])   # idempotent apply
            item.state = "done"
        else:
            self._fail(item, r["type"], max_attempts)
    for cid, item in expected.items():
        if cid not in reported:             # whole-job expiry reports nothing
            self._fail(item, "missing from results", max_attempts)

Worked example

Twelve ticket summaries go out in batches of at most five. Ticket 7 is malformed and will always error. The first batch hits the end of its window with half its requests done; later batches finish. The ledger's results from the example's tests:

The end state is the same, which is the point of the ledger: the provider's behaviour changes how much is resubmitted, not whether the system converges. Two more tests matter in production. Reconciling the same result file twice must change nothing, because operators re-run reconciliation after crashes. And a late success from an earlier attempt must not be applied on top of the newer attempt; the example ignores it, which is simple but pays for both. Accepting the late result and cancelling the newer attempt is the cheaper alternative where the provider supports cancellation.

Deadlines and fallbacks

A completion window is not a deadline you control. If a nightly evaluation must be complete by 07:00 for a release decision, submit early, and at a cutoff send whatever is still pending through the synchronous API at full price. Record the fallback share as a metric: if it creeps up, either the submission time or the batch size is wrong. Keep batches smaller than the maximum, because the unit of loss on whole-job expiry is the job.

Verification

• Count items per state before and after every reconcile; pending plus submitted plus done plus review must equal the total.
• Alert on unknown custom_ids in result files; they mean a ledger bug or a stale file.
• Track resubmission rate, review count and fallback share per job.
• Compare the bill with the ledger: billed requests should equal succeeded results plus whatever the provider documents as billable on expiry.

Failure modes

• Matching results by line order, which works in tests and fails on the first reordered file.
• Reusing the same custom_id on resubmission, so an old result is applied as if it answered the new request.
• Treating a whole-job expiry as success because the results list is empty rather than missing.
• Resubmitting invalid requests for ever, paying for the same error every night.
• Batching the steps of one agent loop and discovering the release gate now takes days.

Practice and related reading

Further reading:

• LLM pricing models: input, output, cache, batch and flex tiers
• Batch inference on your own accelerators
• LLM spend alerts: metering and burn rates in dollars

Sources and scope

Checked against these sources on 21 October 2026. Examples and figures are illustrative, not measurements.

• OpenAI API documentation: Batch API
• Claude API documentation: Batch processing (Message Batches API)
• Gemini API documentation: Batch Mode