Why architecture matters here
Rate limiting sounds like one algorithm and is actually a stack of decisions. Where to enforce (edge or service). What to key by (user, IP, org, route). What algorithm (token bucket, sliding window, leaky bucket). How to share state across replicas. How to respond. How to observe. How to change policies safely.
The architecture matters because the fast path is critical — every request goes through it. Yet correctness matters too — a wrong shared counter means either over-limit letting through or under-limit blocking. The combined design has a local hot path and a shared truth.
With this plane, you can add new tiers, tune abuse detection, and rebalance without hurting p50 latency.
The architecture: every piece explained
The top strip is the request path. Ingress edge is where enforcement lives — an envoy sidecar, a gateway, or a shared library. Local bucket is an in-memory token bucket for micro-second decisions; refills happen on a timer. Distributed store — Redis or DynamoDB — is the shared truth for cross-replica correctness; the local bucket syncs with it. Sliding window is the smoothing algorithm: instead of fixed windows that snap at boundaries, a rolling counter or a log-based window gives fair enforcement across the boundary.
The middle row is the policy shape. Key strategy decides fairness — per user, per IP, per org, per route, or a composite. Policy config holds the limits, bursts, and tier overrides (free vs pro vs internal). Response mode is how you tell the client — HTTP 429 with Retry-After and a JSON body describing the tier. Anti-abuse adds triggers for burst patterns and temporarily lowers a limit or challenges with a CAPTCHA.
The bottom rows are operations. Observability exports per-key hits, blocks, and latency; dashboards must show "how many users are within 10% of their limit right now" so tuning stays real. Fallback encodes the fail-mode: fail-open (allow when Redis is down) or fail-closed (block); the right choice depends on threat model. Deploy pipeline puts policy changes through canary and staged rollout with rollback plans.
End-to-end flow
End-to-end: a user with a 100 requests/minute free tier sends bursts. Each request hits the edge; the edge consults the local bucket, decrements a token, and forwards. The local bucket periodically reconciles with the Redis counter for that user key; a small drift is expected but bounded. When the user exceeds the limit, the bucket returns empty, the edge returns 429 with Retry-After: 42s. The sliding window ensures no snap-back at the minute boundary — a user who spent 50 tokens in the last 30 seconds cannot instantly claim 100 more at the next minute mark. Observability shows 0.4% of requests blocked, 12% of users at 80%+ of limit — healthy. When policy config changes limits, it rolls to 1% of edges, monitored for block-rate change, then to 100%.