Pacing an Agent Fleet From Provider Rate-Limit Headers: Request and Token Limits, Reset Times, 429s and Shared Budgets

Pace from what the provider tells you, not from guesses

A fleet of agents sharing one API key will, sooner or later, send more than the provider allows in a minute. The naive response is to hit the limit, take the 429, and retry. That works for one client and collapses for a fleet: many clients retrying at once synchronise into a thundering herd, and the provider keeps returning 429s because the retries themselves are the overload. The better approach is to pace: use documented rate-limit metadata when present and slow down before refusal. Missing headers, transport failures and concurrent clients still require conservative fallback and bounded retries.

What the headers actually say

All three major providers return rate-limit metadata on responses, with the same shape and different names. The key realisation is that limits are measured separately for requests and for tokens, and the token limit is usually the one you hit first.

Two provider-specific details matter. Anthropic uses a token-bucket model, so capacity replenishes continuously rather than resetting on a fixed boundary, and on most models only uncached input tokens count toward the input-token limit, so a high cache hit rate raises your effective throughput. The remaining-token headers are rounded to the nearest thousand and reflect the most restrictive limit currently in effect. OpenAI reports its reset as a duration string such as 6m0s rather than a timestamp.

A pacer that reads the headers

The pacer keeps the last seen remaining counts and reset times, and before each request it decides whether to wait. Headers can reveal limits before a 429, but missing or invalid metadata must not become unlimited capacity. Network errors may provide no response at all.

Illustrative python, checked for syntax; adapt names and storage to your system.

import time

class Pacer:
    """Shared across a fleet: slow down before the provider refuses."""

    def __init__(self, floor=0.05):
        self.req_remaining = None
        self.tok_remaining = None
        self.reset_at = None
        self.floor = floor          # fraction at which we start throttling

    def observe(self, headers):
        self.req_remaining = int(headers.get("anthropic-ratelimit-requests-remaining", 0))
        self.tok_remaining = int(headers.get("anthropic-ratelimit-tokens-remaining", 0))
        self.reset_at = headers.get("anthropic-ratelimit-tokens-reset")

    def should_wait(self, limit):
        """Return seconds to wait before the next call, or 0 to proceed."""
        if self.tok_remaining is None:
            return 0
        if self.tok_remaining <= self.floor * limit and self.reset_at:
            delay = iso(self.reset_at) - time.time()
            return max(0, delay)
        return 0

The pacer uses the token headroom, not the request headroom, because tokens are usually the binding limit. When remaining tokens fall below a floor, it waits until the reset time the provider reported rather than guessing. The runnable example feeds synthetic header sequences through this pacer and shows it holding back before the limit instead of after. Nothing in it calls a provider.

The 429 you must not retry

Not every 429 means slow down and try again. Anthropic returns a spend-cap error with type rate_limit_error and error.details.error_code equal to enforced_spend_limit_reached, and this response carries no retry-after header; retrying it, including the SDK automatic retries, fails until access resumes on the stated date. A 529 overloaded_error is different again: it means the API is temporarily overloaded across all users, and it is worth a backed-off retry. The rule is to branch on the error details: honour retry-after when present, back off with jitter on a 429 or 529 that is genuinely transient, and stop entirely on a spend cap or a quota error that needs human action rather than a retry.

A worked example

Ten agents share a key with a 2,000,000 input-token-per-minute limit. Each averages 40,000 input tokens per call. Without a pacer they can collectively start sixty calls in a few seconds, offer 2.4M tokens, cross the limit, and take a wave of 429s that then retry together. With the shared pacer, each client sees the remaining-token header fall and begins waiting once it drops below five percent of the limit, so the fleet lands just under 2.0M tokens for the minute and takes no 429s. Because cached input tokens do not count on most Anthropic models, raising the cache hit rate to eighty percent lets the same fleet process far more total tokens before the pacer engages, which is a cheaper fix than asking for a higher limit.

Jitter and a shared view

Two details decide whether pacing scales past one process. The first is jitter: when many clients do have to back off, they must not all wake at the same instant, or they re-synchronise into the next herd. Add a random offset to each wait so the fleet spreads its retries across the window rather than firing them together. The second is where the remaining counts live. A client that paces only on its own last response has a stale, partial view of a limit that the whole fleet is spending, so it consistently overshoots. Keeping the latest remaining-token value and reset time in a small shared store that every client reads before a call turns many independent guesses into one coordinated budget. The shared store does not need to be exact or transactional; a value a second or two old is enough to keep the fleet below the limit, because the provider replenishes continuously and the floor gives headroom. The cost of getting this wrong is not only the 429s themselves but the retries they trigger, which are the load that keeps the limit exceeded, so a pacer that reads the headers is cheaper than any retry policy that waits to be refused.

Failure modes

• Pacing on the request limit while the token limit is the one you hit, so you throttle too late.
• Retrying a spend-cap 429 in a loop; check error_code and stop instead.
• Parsing the OpenAI reset as a timestamp when it is a duration string such as 6m0s.
• Each client pacing on its own view with no shared state, so the fleet still overshoots; share the remaining counts.
• Ignoring that cached tokens do not count toward the Anthropic input-token limit, and so over-throttling a cache-heavy workload.

Verification

Drive the pacer with recorded header sequences, including one that walks remaining tokens down to zero, and assert it waits before the count reaches zero rather than after. Add a test that feeds a spend-cap error body and asserts the pacer stops rather than schedules a retry, and one that feeds a 529 and asserts a backed-off retry. In staging, run the fleet against the real API at a modest limit and confirm the 429 count stays at zero while throughput sits just under the limit. Keep the minimum observed remaining-token value per minute on a dashboard; if it regularly reaches zero, the floor is too low or the fleet is too large for the limit.

Practice and related reading

Further reading:

• Rate limiting
• Retry and backoff
• Connection pooling
• LLM error budgets

Sources and scope

Checked against these sources on 23 October 2026. Examples and figures are illustrative, not measurements.

• Anthropic API: Rate limits (anthropic-ratelimit-* headers, cache-aware ITPM)
• OpenAI API: Rate limits (x-ratelimit-* headers)
• Google Gemini API: Rate limits (RPM, TPM, RPD)
• Anthropic: Building effective agents