Why architecture matters here
Consider what happens without a gateway. Each of forty services terminates its own TLS, parses its own JWTs, and implements its own rate limiting — badly, in four languages. Security reviews multiply by forty. A leaked partner credential must be revoked in every service's config. Mobile clients hard-code forty hostnames, so repartitioning a service behind the scenes becomes a client-release problem. The migration from REST to gRPC, or v1 to v2 auth, becomes a distributed campaign with no chokepoint to measure progress. Centralizing these at a gateway is not elegance; it is the only way policy changes stay O(1) instead of O(services).
But centralization concentrates risk in exact proportion. The gateway's availability multiplies into everything: 99.9% gateway uptime caps every API at 99.9% no matter how good the services are. Its latency adds to every request — a well-built data plane adds single-digit milliseconds, but each filter that calls out (auth introspection, remote rate-limit checks) can add tens. And its configuration is a loaded weapon: a wildcard route that shadows a specific one, a mistyped timeout, a filter reordered — each ships instantly to 100% of traffic unless the control plane enforces staged rollout.
The architecture below is shaped by three disciplines that manage this concentration. Keep the hot path local: every per-request decision should resolve from in-memory state (cached JWKS keys, replicated route tables, local token buckets syncing asynchronously) rather than synchronous remote calls. Fail deliberately: every filter declares fail-open or fail-closed ahead of time — auth fails closed, telemetry fails open, rate limiting is a policy choice. Treat config as deploys: versioned, validated, canaried, and instantly revertible.
The architecture: every piece explained
Top row of the diagram: the entry path. An edge/L4 balancer — anycast IPs, TLS termination or passthrough, volumetric DDoS absorption — spreads connections across data-plane replicas. The replicas are identical and stateless with respect to requests; all their knowledge (routes, certificates, policies) is pushed configuration, which is what lets them scale horizontally and restart freely. This statelessness is the gateway's most load-bearing property: preserve it ruthlessly.
Middle row: the filter chain each request walks. The router matches host, path, method, and headers against the route table — longest-prefix or priority-ordered — and the matched route selects everything downstream: which filters run, with what parameters, to which upstream. The authentication filter validates the bearer token: signature against a JWKS keyset cached in memory and refreshed in the background, then claims-based authorization (scopes, audience) per route policy; opaque tokens needing introspection get a short local cache to keep the IdP off the hot path. The rate limiter enforces token buckets per API key, user, or IP — local buckets with async synchronization to Redis give low latency with approximate global limits, while strict-consistency limits pay a remote check per request. Transforms — header injection (correlation ids, authenticated user context), path rewrites, protocol translation (HTTP/JSON to gRPC), response shaping — run last, and are the chain's most common source of accidental complexity.
Bottom rows: getting to the service and staying honest. The resilience layer applies per-route timeouts, retry policies with budgets (retry only idempotent methods, cap total retries fleet-wide), and circuit breakers that eject upstreams exceeding error thresholds. Upstream pools hold warm connections per service with active health checks, load-balancing policies (round-robin, least-request, ring-hash for stickiness), and outlier ejection. The control plane compiles the route registry from service teams' declarations (often Kubernetes CRDs or a git repo), validates, versions, and pushes it — Envoy's xDS being the canonical protocol — with canary staging. Observability emits per-route RED metrics, access logs, and trace spans; the gateway is the one place that sees everything, which makes its telemetry the incident room's first tab.
End-to-end flow
Follow POST /v2/orders from a mobile client. DNS resolves to the edge's anycast IP; TLS terminates at the balancer (mTLS onward to the gateway); the connection lands on replica 14 of 60. The router matches api.example.com + /v2/orders + POST to route orders-v2-write, which declares: JWT auth with scope orders:write, rate limit 50 rps per API key, a header transform, upstream orders-svc, timeout 2s, no retries (non-idempotent).
The auth filter verifies the JWT's signature against the cached JWKS keys (a memory operation; the cache refreshed from the IdP ninety seconds ago), checks expiry, audience, and the required scope, and stamps X-User-Id and X-Request-Id onto the request while stripping any client-supplied copies of those headers — the services behind the gateway trust these headers precisely because the gateway owns them. The rate limiter debits the local token bucket for this API key (background-synced with Redis so the key's global rate holds approximately across 60 replicas); tokens remain, so the request proceeds — otherwise a 429 with Retry-After would end it here, cheaply, before any upstream work.
The resilience layer starts the 2s timeout and hands the request to the orders-svc pool, where least-request balancing picks the healthiest of 12 endpoints over a warm HTTP/2 connection. The service answers 201 in 80ms; the gateway streams the response back, appends nothing (response transforms are off for this route), and emits: an access log line, RED metrics tagged route=orders-v2-write, and a trace span linking the client's journey to the service's own spans. Total gateway overhead: about 2ms. The interesting story is the counterfactuals — had the service hung, the 2s timeout would have returned a 504 rather than letting mobile clients queue; had 3 of 12 endpoints started erroring, outlier ejection would have quietly removed them; had a config push ten minutes earlier broken this route, the canary stage should have caught it on 1% of traffic.