This page is superseded by API gateway architecture — one front door for auth, routing, rate limits, and shaping, which covers the same subject in more depth. It is kept so existing links keep working.
Why architecture matters here
Consider what happens without a gateway. Each of forty services terminates its own TLS, parses its own JWTs, and implements its own rate limiting — badly, in four languages. Security reviews multiply by forty. A leaked partner credential must be revoked in every service's config. Mobile clients hard-code forty hostnames, so repartitioning a service behind the scenes becomes a client-release problem. The migration from REST to gRPC, or v1 to v2 auth, becomes a distributed campaign with no chokepoint to measure progress. Centralizing these at a gateway is not elegance; it is the only way policy changes stay O(1) instead of O(services).
But centralization concentrates risk in exact proportion. The gateway's availability multiplies into everything: 99.9% gateway uptime caps every API at 99.9% no matter how good the services are. Its latency adds to every request — a well-built data plane adds single-digit milliseconds, but each filter that calls out (auth introspection, remote rate-limit checks) can add tens. And its configuration is a loaded weapon: a wildcard route that shadows a specific one, a mistyped timeout, a filter reordered — each ships instantly to 100% of traffic unless the control plane enforces staged rollout.
The architecture below is shaped by three disciplines that manage this concentration. Keep the hot path local: every per-request decision should resolve from in-memory state (cached JWKS keys, replicated route tables, local token buckets syncing asynchronously) rather than synchronous remote calls. Fail deliberately: every filter declares fail-open or fail-closed ahead of time — auth fails closed, telemetry fails open, rate limiting is a policy choice. Treat config as deploys: versioned, validated, canaried, and instantly revertible.
The architecture: every piece explained
Top row of the diagram: the entry path. An edge/L4 balancer — anycast IPs, TLS termination or passthrough, volumetric DDoS absorption — spreads connections across data-plane replicas. The replicas are identical and stateless with respect to requests; all their knowledge (routes, certificates, policies) is pushed configuration, which is what lets them scale horizontally and restart freely. This statelessness is the gateway's most load-bearing property: preserve it ruthlessly.
Middle row: the filter chain each request walks. The router matches host, path, method, and headers against the route table — longest-prefix or priority-ordered — and the matched route selects everything downstream: which filters run, with what parameters, to which upstream. The authentication filter validates the bearer token: signature against a JWKS keyset cached in memory and refreshed in the background, then claims-based authorization (scopes, audience) per route policy; opaque tokens needing introspection get a short local cache to keep the IdP off the hot path. The rate limiter enforces token buckets per API key, user, or IP — local buckets with async synchronization to Redis give low latency with approximate global limits, while strict-consistency limits pay a remote check per request. Transforms — header injection (correlation ids, authenticated user context), path rewrites, protocol translation (HTTP/JSON to gRPC), response shaping — run last, and are the chain's most common source of accidental complexity.
Bottom rows: getting to the service and staying honest. The resilience layer applies per-route timeouts, retry policies with budgets (retry only idempotent methods, cap total retries fleet-wide), and circuit breakers that eject upstreams exceeding error thresholds. Upstream pools hold warm connections per service with active health checks, load-balancing policies (round-robin, least-request, ring-hash for stickiness), and outlier ejection. The control plane compiles the route registry from service teams' declarations (often Kubernetes CRDs or a git repo), validates, versions, and pushes it — Envoy's xDS being the canonical protocol — with canary staging. Observability emits per-route RED metrics, access logs, and trace spans; the gateway is the one place that sees everything, which makes its telemetry the incident room's first tab.
End-to-end flow
Follow POST /v2/orders from a mobile client. DNS resolves to the edge's anycast IP; TLS terminates at the balancer (mTLS onward to the gateway); the connection lands on replica 14 of 60. The router matches api.example.com + /v2/orders + POST to route orders-v2-write, which declares: JWT auth with scope orders:write, rate limit 50 rps per API key, a header transform, upstream orders-svc, timeout 2s, no retries (non-idempotent).
The auth filter verifies the JWT's signature against the cached JWKS keys (a memory operation; the cache refreshed from the IdP ninety seconds ago), checks expiry, audience, and the required scope, and stamps X-User-Id and X-Request-Id onto the request while stripping any client-supplied copies of those headers — the services behind the gateway trust these headers precisely because the gateway owns them. The rate limiter debits the local token bucket for this API key (background-synced with Redis so the key's global rate holds approximately across 60 replicas); tokens remain, so the request proceeds — otherwise a 429 with Retry-After would end it here, cheaply, before any upstream work.
The resilience layer starts the 2s timeout and hands the request to the orders-svc pool, where least-request balancing picks the healthiest of 12 endpoints over a warm HTTP/2 connection. The service answers 201 in 80ms; the gateway streams the response back, appends nothing (response transforms are off for this route), and emits: an access log line, RED metrics tagged route=orders-v2-write, and a trace span linking the client's journey to the service's own spans. Total gateway overhead: about 2ms. The interesting story is the counterfactuals — had the service hung, the 2s timeout would have returned a 504 rather than letting mobile clients queue; had 3 of 12 endpoints started erroring, outlier ejection would have quietly removed them; had a config push ten minutes earlier broken this route, the canary stage should have caught it on 1% of traffic.