Why architecture matters here
The reason rate limiting is architectural is that the quotas it respects are not advisory — they are hard external constraints that determine whether your agent works at all. A model provider that returns 429 when you exceed its tokens-per-minute limit is not negotiating; it is refusing service, and an agent that blindly retries into that refusal only deepens the throttle and can get the account rate-limited harder or suspended. The dependency's capacity is fixed and outside your control, so the only lever you have is shaping your own outbound traffic to fit under it. That shaping cannot be an afterthought bolted onto individual call sites; it has to be a coherent layer, because the quota is shared across every call your fleet makes and only a coordinated limiter can keep the aggregate within bounds.
Rate limiting matters because it converts uncontrolled burstiness into controlled flow. Agent workloads are spiky by nature — user traffic arrives in waves, a single agent turn can fan out into many tool calls, and retries amplify load exactly when a dependency is already stressed. Without a limiter, these spikes hit the provider at full amplitude, cross the quota, and trigger throttling that makes everything worse. The token bucket absorbs the spike into its capacity and releases it at the sustainable refill rate, so the provider sees a smooth stream it can serve rather than a burst it must reject. The limiter is what stands between your fleet's natural burstiness and the provider's intolerance of it.
The second load-bearing property is fairness and isolation across tenants. A platform serving many tenants or many agents from one shared provider quota faces a noisy-neighbor problem: one tenant's spike can consume the entire quota and starve everyone else, turning one customer's burst into a fleet-wide outage. Per-key token buckets — one bucket per tenant, per agent, or per API key — partition the quota so each consumer gets a bounded, guaranteed share and cannot cannibalize the rest. This isolation is what makes a shared-quota multi-tenant platform viable: the limiter enforces the boundary that keeps one heavy user from degrading service for all the others, which is a correctness property of the platform, not a mere optimization.
The third reason is that the limiter is where backpressure becomes explicit and manageable. When demand exceeds the sustainable rate, something has to give, and the limiter makes that a deliberate, observable decision rather than an implicit collapse. It can block callers (applying backpressure that slows the intake), reject with a clear signal so upstream can queue or shed, or shape into a queue that drains at the refill rate. Each choice has consequences — blocking ties up threads, rejecting pushes the problem upstream — but they are choices you get to make and instrument, at one chokepoint, instead of discovering the system's overload behavior during an incident. On the Java platform, where virtual threads make blocking cheap, the limiter's backpressure strategy interacts directly with the concurrency model, which is one more reason it deserves to be a designed layer rather than scattered guards.
The architecture: every piece explained
Top row: where the limiter sits. An agent call — a model invocation or a tool call — does not go straight to the dependency; it passes through the rate limiter, which holds a token bucket keyed by whatever dimension you are governing (the provider, the tenant, the API key). Only if the limiter grants a permit does the call reach the provider or tool, which has its own hard quota (requests-per-minute, tokens-per-minute, calls-per-second). The limiter's job is to ensure the aggregate rate of granted permits stays under that quota so the dependency never sees traffic it will reject.
Middle row: the bucket mechanics. The bucket has two parameters that fully define its behavior: a capacity (the maximum tokens it can hold, which sets the largest allowed burst) and a refill rate (tokens added per unit time, which sets the sustained long-run rate). To make a call, the code must acquire a permit — remove a token — and if the bucket is empty it either blocks until a token refills or is rejected immediately, per the configured policy. For token-based model quotas, a call may need to remove a number of tokens proportional to its estimated token cost, not just one, so the bucket meters actual consumption. When the limiter must span multiple JVM instances, the bucket state lives in a distributed store — commonly Redis with an atomic script — so all instances share one consistent count rather than each enforcing its own local limit.
Bottom rows: the surrounding behavior. 429 handling closes the loop with reality: even a well-tuned limiter can occasionally overshoot, so when the provider returns 429 the client backs off, honors any Retry-After header, and ideally feeds that signal back to tighten the bucket adaptively. Fairness is achieved with per-tenant or per-key buckets so one consumer cannot starve others. The ops strip names the tuning surface: choosing capacity and refill to match the provider quota with headroom, deciding between a local per-instance limiter and a distributed shared one, selecting the backpressure behavior (block versus reject), and instrumenting the limiter so permit waits, rejections, and 429s are all visible.
The choice between a local and a distributed limiter is the defining architectural decision, and it is a genuine trade-off rather than a clear win either way. A local token bucket in each JVM is simple, fast, and dependency-free — but N instances each enforcing a limit of R means the fleet's real limit is N×R, so you must divide the provider quota by the instance count and re-tune every time you scale, and autoscaling makes that a moving target that easily overshoots. A distributed bucket backed by a shared store enforces one true fleet-wide limit regardless of instance count, which is correct under autoscaling, but adds a network round-trip and a hard dependency to the hot path of every call — and if that store is slow or unavailable, every call is affected. Mature deployments often blend the two: a distributed limiter for the authoritative fleet-wide quota plus a local limiter as a fast first cut and a fail-safe if the store is unreachable. Deciding this consciously, with the scaling behavior in mind, is what separates a limiter that holds under load from one that either throttles needlessly or lets the fleet blow its quota the moment it scales out.
End-to-end flow
Trace a burst of agent calls under a distributed token-bucket limiter. A provider quota of 600 requests per minute is modeled as a bucket with a refill of 10 tokens per second and a capacity of 60 — allowing a burst of up to 60 calls if the bucket is full, then settling to 10 per second sustained. The bucket state lives in Redis, shared across all agent instances.
Steady state. Under normal load the fleet makes roughly 8 calls per second. Each call acquires a permit — an atomic Redis operation that checks the token count, refills based on elapsed time, and decrements if a token is available. Since demand is below the 10/second refill, tokens are generally available, permits are granted immediately, and the bucket hovers near full. Calls flow straight through to the provider, well under quota.
The burst. A wave of user requests arrives and the fleet suddenly wants to make 100 calls in a second. The first roughly 60 calls find tokens in the full bucket and are granted immediately — this is the controlled burst the capacity permits, and it lets the platform respond snappily to the spike. The bucket now empties. The remaining calls find it empty and, per the configured blocking policy, wait; on Java virtual threads this blocking is cheap, so thousands of waiting calls do not exhaust a thread pool. As the bucket refills at 10 tokens per second, waiting calls are granted at that rate, draining the backlog smoothly. The provider therefore sees an initial burst of 60 followed by a steady 10 per second — comfortably within its 600/minute quota — instead of a raw 100-in-one-second spike that would have blown the limit and triggered throttling.
Overshoot and recovery. Suppose that despite the limiter a transient miscount, or another client sharing the same provider account, pushes actual traffic over the quota and the provider returns a 429 with Retry-After: 2. The ADK-Java client catches the 429, does not blindly retry, waits the two seconds the provider asked for, and retries with backoff; an adaptive limiter also momentarily tightens the bucket in response to the 429 signal, reducing pressure until the provider recovers. Now the multi-tenant angle: because the fleet uses per-tenant buckets, a single tenant that generated this burst drew only from its own bucket. Once that tenant's bucket emptied, its further calls waited, but other tenants — with their own buckets still full — continued at full speed, entirely unaffected by the noisy neighbor. The provider quota was respected in aggregate, the bursty tenant was smoothed rather than failed, and the quiet tenants never noticed. That combination — burst tolerance, sustained-rate enforcement, graceful 429 handling, and per-tenant isolation — is the whole reason the token bucket, rather than a naive fixed-window counter, is the right shape for an agent platform's traffic.