Why architecture matters here
Rate limiting matters at three levels: cost, reliability, and fairness. Cost because a runaway client burns budget in seconds. Reliability because unrestricted traffic saturates backends. Fairness because one aggressive tenant should not degrade service for others. Each of these has a different remedy, and the architecture must express all three.
The math is simple, the engineering is not. A single counter in Redis works for a small service, but at scale you need sharded counters, atomic scripts, and fallbacks for when Redis is briefly unavailable. Naive implementations either allow bursts that overwhelm the backend, or reject legitimate traffic during transient issues, or both. The right architecture handles both bursts and hot shards without becoming a bottleneck itself.
Signaling matters as much as enforcement. Clients that get 429 with no Retry-After header will retry immediately and make the problem worse. Clients that get useful signals back off intelligently. The gateway's response format shapes client behavior; getting it right multiplies the effectiveness of the rate limiter itself.
The architecture: every layer explained
Read the diagram from top to bottom.
Client SDK. A well-behaved SDK respects 429 responses, honors Retry-After, and adds jitter to retries. Older SDKs and hand-written clients often don't; documentation and default behavior in your SDK are your first line of defense against retry storms.
Edge / CDN. The edge layer (Cloudflare, Akamai, Fastly, AWS CloudFront) does per-IP rate limiting for volumetric abuse and known-bad IPs. This shields the origin from bots and volumetric attacks. The edge cannot do fine-grained per-tenant limits because it doesn't have identity — that job lives further inside.
L7 Gateway. The gateway (Envoy, Kong, cloud LB) parses the request, extracts the tenant and user ID from the JWT, and performs the per-tenant rate check. This is where the majority of rate limiting decisions happen.
Local Token Bucket. Each pod maintains an in-memory token bucket per tenant per endpoint. The bucket refills at the configured rate; each request consumes a token. If tokens remain, allow; otherwise, check the global limit. This local check is O(1) and adds ~1 microsecond of latency; it soaks up 90%+ of requests without touching Redis.
Global Sliding Window. For requests that pass the local check or need cross-pod visibility, the gateway increments a global counter in Redis. Sliding window (as opposed to fixed window) uses two adjacent windows and a weighted sum to smooth burst behavior at window boundaries. A Lua script does the read-modify-write atomically so concurrent requests do not overshoot the limit.
Redis Cluster. A Redis cluster holds the counters. Keys are shaped like "rl:{tenant}:{endpoint}:{window}" and sharded by tenant. High-traffic tenants may be on dedicated shards to prevent hot-key congestion. Persistence is optional; rate-limit counters are safe to lose (they simply reset to zero) if the tradeoff is acceptable.
Lua scripts. The atomic increment-and-check runs as a Lua script on Redis. This avoids race conditions where two concurrent gateway processes each see a value below the limit and both allow, ending up above. Lua adds ~50 microseconds of latency per check but is essential for correctness.
Fallback. When Redis is unavailable (network partition, restart, congestion), the gateway falls back to local-only limits. Two failure modes are worth thinking about: fail-open (allow requests, accepting risk) or fail-closed (deny requests, accepting user friction). Most APIs choose fail-open with a stricter local budget.
Policy engine. Rate limits are not one-size-fits-all. Free-tier tenants get 60 requests per minute. Paid tiers get 6,000. Enterprise gets a custom quota. Burst allowances let short spikes through without hitting the sustained rate. All of this lives in a policy config that is versioned in git and hot-reloadable.
Response signaling. A 429 response includes Retry-After (seconds to wait), X-RateLimit-Limit (the ceiling), X-RateLimit-Remaining (current available), and X-RateLimit-Reset (Unix time when the window resets). Well-behaved clients use these to schedule retries intelligently.
Metrics and audit. Every allow and every shed is counted per tenant, per endpoint. Dashboards show the ratio; alerts fire when a tenant approaches its quota. Audit logs record every 429 so support can reconstruct what a customer saw.
End-to-end request flow
Trace a request. A paid-tier tenant with 6,000 requests-per-minute limit sends a burst of 500 requests in 3 seconds. The first 350 requests hit different gateway pods; each pod's local token bucket allows them because each pod locally sees only a fraction of the traffic. The 351st request from that tenant triggers a global check on Redis.
The Redis Lua script increments the sliding-window counter, sees the tenant is at 351 requests in the current minute, checks against the 6,000/min limit, and allows. The gateway proceeds to serve. The counter now shows 351.
Requests continue. At request 6,001 the counter check returns "over limit." The gateway responds with 429, Retry-After: 45 (seconds until the window rolls). The client SDK reads the header, backs off, and resumes at 45 seconds.
Meanwhile the local buckets are also updated based on Redis's authoritative view — a background process syncs the "remaining" back down to each pod, so the next few requests hit local-limit before touching Redis. This keeps latency low even during rate-limit events.
If Redis has a brief outage during the burst, the gateway falls back to local limits. Local limits are conservative (say, 500/min per pod × 10 pods = 5,000/min effective) to prevent overrun. When Redis recovers, the gateway resumes global checks and the tenant sees consistent behavior.