Rate limiting architecture, in depth: rate versus concurrency limits, adaptive limits and outbound throttling

Two kinds of limit, and why a rate limit is not enough

Start from Little's law, which holds for any stable queueing system: the average number of items inside equals the arrival rate multiplied by the average time each spends inside. For a service, in_flight = requests_per_second x latency_seconds. At 400 requests per second and 50 ms, 20 requests are in flight. If the database behind the service slows to 500 ms, the same 400 requests per second means 200 in flight, and no client changed anything.

That is why a rate limit alone cannot protect a service. It caps the arrival rate, but threads, pool slots and buffers are consumed by in-flight work, the product of rate and latency. A concurrency limit caps the product directly: when latency rises it admits fewer new requests and rejects the rest immediately, while the service is still healthy enough to answer quickly.

Four admission points

The diagram above shows the four admission points in request order.

Edge. The CDN or load balancer applies coarse per-IP and per-network limits to absorb floods before they cost TLS handshakes and application CPU. The numbers are generous; the goal is to stop obvious abuse, and the edge often cannot see the API key anyway.

Gateway. After authentication the gateway knows the tenant and key, so contractual quotas live here: 1,000 requests per minute for the free tier, separate budgets for expensive routes. The counters must agree across replicas; algorithms and the Redis implementation are covered in Rate limiter architecture: GCRA, leases and failure policy, and requirements and shadow rollout in Rate limiter design: requirements to rollout. In short, a token bucket gives an explicit burst allowance and a sliding window counter reads like "N per minute" to customers; both need a few bytes per key.

Service. Each process keeps a concurrency limit in front of its handlers, with no shared state: every process protects its own threads and pools. This is the layer most systems are missing.

Outbound client. Every call to a dependency passes through a client-side limiter that enforces the dependency's rate, caps your in-flight calls to it, and reacts to its 429 and 503 answers. Without it, your retries become the flood the dependency has to absorb.

Two kinds of limit, and why a rate limit is not enough

Start from Little's law, which holds for any stable queueing system: the average number of items inside equals the arrival rate multiplied by the average time each spends inside. For a service, in_flight = requests_per_second x latency_seconds. At 400 requests per second and 50 ms, 20 requests are in flight. If the database behind the service slows to 500 ms, the same 400 requests per second means 200 in flight, and no client changed anything.

That is why a rate limit alone cannot protect a service. It caps the arrival rate, but threads, pool slots and buffers are consumed by in-flight work, the product of rate and latency. A concurrency limit caps the product directly: when latency rises it admits fewer new requests and rejects the rest immediately, while the service is still healthy enough to answer quickly.

Four admission points

The diagram above shows the four admission points in request order.

Edge. The CDN or load balancer applies coarse per-IP and per-network limits to absorb floods before they cost TLS handshakes and application CPU. The numbers are generous; the goal is to stop obvious abuse, and the edge often cannot see the API key anyway.

Gateway. After authentication the gateway knows the tenant and key, so contractual quotas live here: 1,000 requests per minute for the free tier, separate budgets for expensive routes. The counters must agree across replicas; algorithms and the Redis implementation are covered in Rate limiter architecture: GCRA, leases and failure policy, and requirements and shadow rollout in Rate limiter design: requirements to rollout. In short, a token bucket gives an explicit burst allowance and a sliding window counter reads like "N per minute" to customers; both need a few bytes per key.

Service. Each process keeps a concurrency limit in front of its handlers, with no shared state: every process protects its own threads and pools. This is the layer most systems are missing.

Outbound client. Every call to a dependency passes through a client-side limiter that enforces the dependency's rate, caps your in-flight calls to it, and reacts to its 429 and 503 answers. Without it, your retries become the flood the dependency has to absorb.

Sizing a concurrency limit from a load test

Size a fixed limit first; adaptive algorithms need a sensible start and ceiling. Load test one instance while raising offered load. Throughput climbs, then flattens, and past that knee only latency grows. Useful concurrency is peak throughput multiplied by the latency at the knee.

Suppose one instance peaks at 600 requests per second with a 40 ms median at the knee. Useful concurrency is 600 x 0.04 = 24; requests beyond that only wait. A limit of 32 leaves headroom for variance and still rejects the excess long before a 200-thread pool is exhausted. A minimal limiter never waits:

import threading

class ConcurrencyLimiter:
    """Admit at most `limit` requests in flight; reject instead of queueing."""

    def __init__(self, limit: int):
        self.limit = limit
        self.in_flight = 0
        self._lock = threading.Lock()

    def try_acquire(self) -> bool:
        with self._lock:
            if self.in_flight >= self.limit:
                return False
            self.in_flight += 1
            return True

    def release(self) -> None:
        with self._lock:
            self.in_flight -= 1

limiter = ConcurrencyLimiter(limit=32)

def handle(request):
    if not limiter.try_acquire():
        return Response(503, headers={"Retry-After": "1"})  # cheap and immediate
    try:
        return do_work(request)
    finally:
        limiter.release()

The rejection costs microseconds, so the caller can try another instance. An unbounded queue instead turns overload into latency for everyone; if you must queue, keep it to a few requests and drop anything past its caller's deadline. Choosing which requests to drop is covered in Load shedding: dropping work to survive overload.

Adaptive limits: AIMD and gradient

A fixed limit is right only for the latency you measured; a 30% slower release makes it too high, bigger instances make it too low. Adaptive limiters adjust from signals the process already sees. Netflix's open-source concurrency-limits library for Java implements the two common families, and Envoy ships an adaptive concurrency filter built on the gradient idea.

AIMD (additive increase, multiplicative decrease, as in TCP congestion control) adds one after successes and multiplies by about 0.9 on a timeout or overload error; it only reacts after something failed. Gradient limiters compare recent latency with a slow baseline and shrink the limit in proportion as soon as queueing starts.

class GradientLimit:
    """Shrink the limit when recent latency exceeds the long-term baseline.

    Call on_sample once per sampling window (say every second) with that
    window's average latency, not once per request: at one call a second the
    0.99/0.01 baseline averages roughly the last 100 seconds.
    """

    def __init__(self, initial=20, min_limit=4, max_limit=200, smoothing=0.2):
        self.limit = float(initial)
        self.min_limit, self.max_limit = min_limit, max_limit
        self.smoothing = smoothing
        self.baseline_ms = None

    def on_sample(self, latency_ms: float, in_flight: int, dropped: bool) -> int:
        if self.baseline_ms is None:
            self.baseline_ms = latency_ms
        self.baseline_ms = 0.99 * self.baseline_ms + 0.01 * latency_ms
        if dropped:
            target = self.limit * 0.9
        else:
            gradient = max(0.5, min(1.0, self.baseline_ms / latency_ms))
            target = self.limit * gradient + self.limit ** 0.5
            if in_flight < self.limit / 2:
                target = min(target, self.limit)   # unused: may shrink, never grow
        self.limit = (1 - self.smoothing) * self.limit + self.smoothing * target
        self.limit = max(self.min_limit, min(self.max_limit, self.limit))
        return int(self.limit)

Three details matter more than the formula. The baseline must move slowly, or a sustained slowdown becomes normal. The limit must not grow while unused. And it needs a floor and ceiling from your static sizing. Export the current limit as a metric: a limit stuck at its floor is a dependency incident nobody has paged on yet.

Limiting your own outbound calls

Your service is someone else's caller. If a provider allows 200 requests per second and you run 20 instances, a limiter of 10 per second per instance keeps you inside the contract without a shared store. Add a concurrency cap per dependency too, so a slow dependency takes a bounded slice of your threads: the bulkhead pattern at the client.

On a 429 or 503, an immediate retry adds load at the worst moment. Honour Retry-After, otherwise back off exponentially with full jitter, and spend retries from a budget:

import random, time

class RetryBudget:
    """Allow retries up to `ratio` of recent first attempts (e.g. 10%)."""
    def __init__(self, ratio=0.1):
        self.ratio, self.attempts, self.retries = ratio, 0, 0
    def can_retry(self) -> bool:        # decay both counters on a timer in real code
        if self.retries < self.ratio * max(self.attempts, 10):
            self.retries += 1
            return True
        return False

def call_with_backoff(send, budget, max_tries=4, base_s=0.1, cap_s=5.0):
    for attempt in range(max_tries):
        if attempt == 0:
            budget.attempts += 1
        resp = send()
        if resp.status not in (429, 503):
            return resp
        if attempt == max_tries - 1 or not budget.can_retry():
            return resp
        retry_after = resp.headers.get("Retry-After")
        if retry_after and retry_after.isdigit():
            delay = float(retry_after)
        else:
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))  # full jitter
        time.sleep(delay)

The budget prevents retry amplification: with three retries per request and no budget, a dependency rejecting most traffic sees up to four times normal load; with a 10% budget, at most 10% extra. How overload signals travel upstream through a chain of services is the subject of Backpressure architecture.

Telling callers what happened

HTTP 429 Too Many Requests is defined in RFC 6585; 503 fits capacity rejection. The Retry-After field, defined in RFC 9110, carries seconds or an HTTP date. Add a machine-readable body naming which limit was hit, since a per-route limit and an exhausted monthly quota need different fixes.

For advertising limits before a client hits them, the IETF HTTPAPI working group has a draft, "RateLimit header fields for HTTP", whose recent revisions (draft 11 was published in May 2026) define two fields: RateLimit-Policy for the quota policies and RateLimit for the remaining service. It is still an Internet-Draft and earlier revisions used other field names; many APIs still send X-RateLimit-* headers. If you adopt it, pin the draft version.

Worked example: checkout and a slow payments provider

A checkout service runs 20 instances behind a gateway and calls a payments provider whose contract allows 200 requests per second per merchant account. Normal traffic is 2,400 checkout requests per second across the fleet, 120 per instance. About 6% of checkouts need a payment authorization, so each instance sends 7.2 payments calls per second and the fleet sends 144, inside the contract. Checkouts without a payment take 25 ms; those with one take about 60 ms, 35 ms of it waiting on the provider.

Limits: per-merchant gateway quotas of 300 checkouts per second; a service knee at 700 requests per second and 70 ms, so useful concurrency 49 and a gradient limiter starting at 64 (floor 8, ceiling 100); outbound, 10 payments calls per second and 8 in flight per instance, with a 10% retry budget.

Now the provider degrades and its latency goes from 35 ms to 800 ms. Little's law gives the new in-flight numbers per instance: payments calls 7.2 x 0.8 ≈ 5.8, under the cap of 8; checkouts about 112.8 x 0.025 + 7.2 x 0.825 ≈ 8.8, far under the limit. The danger arrives when the provider also starts answering 429 and impatient clients retry checkout, pushing offered load towards 200 per instance. Payments demand rises to 12 per second per instance, which would need 9.6 calls in flight at 800 ms: the in-flight cap and the 10 per second limit reject the excess immediately, and the retry budget stops those rejections from being retried more than 10% of the time. The provider sees at most its contract rate of 200 per second instead of 240 first attempts plus up to three retries each, nearly 960. In-flight checkouts per instance reach only about 200 x (0.94 x 0.025 + 0.06 x 0.825) ≈ 15, well under the service limit, so the gradient limiter drifts down a little and rejects nothing: here the outbound controls do the protecting, and the service limit is the backstop for a slowdown that hits every request. Dashboards show outbound rejections and provider 429s rising together, pointing straight at the provider.

Failure modes

• Rate limit set, concurrency unbounded. Quotas all green while thread pools are exhausted by a slow dependency. Fix: a concurrency limit per process and per dependency.
• Limiter that queues. A blocking semaphore with no timeout turns rejection into latency. Fix: try-acquire, or a short bounded wait tied to the caller's deadline.
• Adaptive limit with a fast baseline. The baseline chases the slowdown and the limit never drops. Fix: a slow baseline, and alert when the limit sits at its floor or ceiling.
• Retry storms. Per-request retries multiply load during an incident. Fix: retry budgets, jittered backoff, honouring Retry-After, and retrying at one layer only.
• Per-instance outbound limits after scaling. Doubling instances doubles your outbound rate and breaches the dependency's contract. Fix: derive per-instance limits from the current instance count, or use a shared counter for that dependency.

Trade-offs

Decide fail-open or fail-closed per route before the outage, and keep local concurrency limits as the safety net either way. Quotas spanning days or months are a different mechanism; see Designing a quota system.

What to do next

1. Draw your request path and mark which of the four admission points exist today and what each one counts.
2. Load test one instance, find the throughput knee, and compute useful concurrency with Little's law.
3. Add a non-blocking concurrency limiter in front of handlers, starting at about 1.3 times that number.
4. Export in-flight count, current limit, admitted and rejected counts per layer, and alert when the limit sits at its floor.
5. Wrap each dependency client with a rate limit, an in-flight cap and a retry budget; honour Retry-After.
6. Make rejections cheap and explicit: 429 for quotas, 503 for capacity, a body naming the limit.
7. Exempt health checks, then run a game day that slows one dependency tenfold and watch the limits react.