Why architecture matters here
The architecture matters because the caller is autonomous and tireless in a way human traffic never is. A person who hits a slow endpoint waits, gets frustrated, and stops; an agent in a retry loop accelerates, because a failed tool call often looks to the model like a reason to try again — sometimes harder, with a larger query. Without a limiter, a single confused or adversarial agent can drive a tool server's backend into the ground in seconds, and because MCP servers frequently wrap a shared, finite resource (a database, a paid API, a search cluster), that collapse takes down every other agent and every other tenant with it. Rate limiting is the mechanism that converts 'one agent misbehaves' from an outage into a contained, per-client throttle.
It matters because tool cost is non-uniform, so a flat 'N calls per minute' limit is simultaneously too strict and too loose. Count a cheap key lookup and an expensive aggregate query the same, and you either set the limit low enough to protect the backend from the expensive calls (needlessly throttling cheap ones and breaking legitimate reasoning) or high enough to allow normal cheap traffic (leaving the backend exposed to a burst of expensive ones). Cost-aware limiting — weighting each tool by how much load it actually imposes and draining a shared budget accordingly — is what lets a single policy protect the backend without strangling ordinary use. The architecture's job is to make the quota reflect real load, not raw call count.
It matters because the response to a limit is part of the contract with an autonomous client. A well-behaved agent can react correctly to a clear signal: a rejection that says 'you are over quota, retry after N seconds' lets the model back off, and structured backpressure turns a would-be retry storm into orderly pacing. A silent drop, a hung connection, or an opaque 500 does the opposite — the model cannot tell it is being limited, so it retries immediately and amplifies the problem. So the architecture is not only about deciding whether to admit a call but about communicating the decision in a way the caller can act on, which is why a machine-readable retry-after and a distinct rate-limit error class are first-class parts of it.
Finally, it matters because rate limiting is inseparable from multi-tenancy and fairness. A tool server usually serves many agents, sessions, or customers at once, and a limiter that only enforces a global cap lets whichever client is greediest consume the whole budget and starve the rest — the classic noisy-neighbor failure. Keying buckets per client and enforcing a fair share means each tenant gets its guaranteed slice and no one can monopolize the server, so the limiter is also the fairness mechanism that makes shared tool infrastructure viable. That fairness has to hold across a horizontally-scaled fleet, which is why the bucket state cannot live only in one process's memory.
The architecture: every piece explained
Top row: the decision pipeline every call passes through. An agent or client issues a tool call. The limiter first identifies and classifies it — which client or tenant is calling, which tool, and how costly that tool is — because every downstream decision keys on those three facts. It then checks a token bucket: a store with a capacity and a steady refill rate, where each admitted call removes tokens proportional to its cost and the bucket refills over time. If enough tokens are present the limiter admits the call to the real tool; if not, it rejects with a 429-style rate-limit error carrying a retry-after hint. The token bucket is the core because it naturally allows short bursts (up to capacity) while bounding the sustained rate (the refill), which matches how agents actually behave.
Middle row: the pieces that make the core robust. Per-tool weights encode that expensive tools cost more tokens, so a single shared budget protects the backend from heavy calls without over-throttling light ones. A backpressure signal — the retry-after or a 'slow down' response — tells the autonomous caller how to pace itself instead of retrying blindly. A concurrency cap bounds the number of in-flight calls per client independently of the call rate, which matters because a few slow tool calls can pin backend resources even at a modest request rate. And a downstream quota protects the real backend the tools wrap — the database, the paid API — so the limiter is not just shaping request rate but guarding the finite resource underneath.
Bottom-left: the multi-tenant guarantee. Because buckets are keyed per client, the limiter enforces a fair share across tenants: each agent or customer gets its own budget, so one client's loop drains only its own bucket and cannot consume the capacity reserved for others. This is what defeats the noisy-neighbor failure and makes a shared tool server safe to offer to many callers at once.
Bottom-right and ops: graceful degradation and the operational surface. When a client is over budget the limiter degrades gracefully — it rejects the excess cheaply, returning a fast rate-limit error rather than queuing work that overloads the backend or crashing under the flood. The ops strip names what running this in production requires: holding the buckets in a shared store (so the limit is enforced consistently across a horizontally-scaled server fleet, not per-instance), exporting metrics per client and per tool (to see who is hitting limits and why), tuning the limits against real traffic (tight enough to protect, loose enough for legitimate multi-step agents), and alerting on sustained 429s, which signal either an abusive client or a limit set too low for honest work.
End-to-end flow
Follow two clients hitting the same MCP tool server — one well-behaved agent doing legitimate multi-step work, one stuck in a retry loop — and watch the limiter treat them differently.
Identification and classification: each call arrives with credentials or a session identity, so the limiter knows which client sent it and which tool it targets. It looks up the tool's cost weight — a cheap lookup costs one token, an expensive search costs ten — and the client's bucket. The two clients have separate buckets, so from the first call their fates are independent; nothing the looping client does can drain the well-behaved client's budget.
The well-behaved agent: it issues a burst of a few tool calls as it reasons through a task, then pauses to think, then issues a few more. Its bucket has capacity for a burst and refills during the pauses, so every call is admitted — the token bucket's burst allowance is exactly what legitimate bursty reasoning needs. Its concurrency stays low because it waits for each result before the next step, so the concurrency cap is never approached. From this client's perspective the limiter is invisible: it never sees a rejection because it never sustains a rate above its refill.
The runaway loop: the second agent misreads a tool result and begins retrying the same expensive search as fast as it can. Its bucket drains in a few calls — ten tokens each against a bucket that refills slowly — and the next call is rejected with a rate-limit error carrying retry-after: 30. A correctly-built agent would back off; this one keeps retrying, but now every retry is rejected cheaply at the limiter, drawing no backend work at all. Meanwhile its concurrency cap ensures that even the calls admitted before the bucket emptied cannot pile up more than a bounded number of slow searches in flight. The expensive backend the search tool wraps sees only the trickle the refill rate permits, never the flood.
Fleet-wide consistency and observability: because the server is horizontally scaled, the two clients' calls may land on different server instances, so the buckets live in a shared store (a fast key-value store) rather than in per-instance memory — otherwise the looping client would get its full quota on each instance and the limit would multiply by the fleet size. The limiter exports per-client, per-tool metrics, so operators see one client sustaining 429s on the search tool and can decide whether it is an abusive client to cut off or a legitimate workload whose limit needs raising. Throughout, the well-behaved agent kept working, the backend stayed healthy, and the runaway loop was contained to its own bucket — the limiter turned a potential outage into a single client's throttle.