Why architecture matters here
Naive serving — load the model in a web framework, call predict() per request — works right up until it meets real traffic, then fails on three independent axes at once. Economics: an accelerator running batch-size-1 inference sits at 5-15% utilization; you pay for the other 85% to do nothing. At fleet scale this is the difference between a serving bill that tracks revenue and one that dwarfs it. Latency: the model call is only one term in the budget — feature lookup, tokenization or preprocessing, queueing, and cross-zone hops stack on top, and without an architecture that measures each stage separately, p99 regressions are unattributable. Change: models ship weekly or daily, and each new version is a behavioral change deployed to live traffic; without staged rollout and automatic rollback, every deployment is a bet of the whole business on an offline eval.
Architecture is also what contains the failure modes unique to ML. A crashed replica is visible; a model quietly answering with degraded quality because an upstream feature pipeline stalled is not — the HTTP status is 200 and the predictions are garbage. The serving stack is where training-serving skew gets caught (or doesn't): if the online feature transformation drifts from what the model saw in training, accuracy decays with no error anywhere. Only an architecture that treats features, model artifacts, and prediction distributions as monitored, versioned surfaces can turn these silent failures into alerts.
Finally, isolation matters because inference workloads fight each other. A large-batch offline scoring job sharing replicas with interactive traffic will destroy interactive p99; a spiky tenant without admission control will starve the rest. The gateway, router, and autoscaler exist to give every consumer a predictable slice of a fundamentally bursty, expensive resource.
The architecture: every piece explained
Top row — the request path. The API gateway terminates TLS, authenticates, enforces per-tenant quotas, and applies admission control: when the backlog exceeds what the SLO can absorb, it sheds load early with a fast 429 rather than letting every caller time out slowly. The model router resolves a logical endpoint (ranker@prod) to a physical deployment: version pinning, traffic-split weights for canaries, per-variant targeting for A/B tests, and fallback chains (if the heavy model's queue is saturated, route to the distilled variant rather than fail). Because the router owns the name→version binding, deployments and rollbacks are routing changes, not process restarts.
The dynamic batcher is the utilization engine. It queues incoming requests per model and releases a batch when either the batch fills (max_batch_size) or the oldest request's wait hits the timeout (max_queue_delay, typically single-digit milliseconds). Those two knobs trade latency against throughput explicitly; shape-aware servers add bucketing so requests with similar sequence lengths batch together and padding waste stays bounded. The model servers behind it are deliberately boring: a pinned runtime (Triton, TorchServe, vLLM, ONNX Runtime) with the model loaded into device memory, exposing health, readiness, and per-stage metrics. Replicas are cattle; all state lives elsewhere.
Middle row — the supporting cast. The feature service answers online feature lookups from a low-latency store, using the same transformation code as training to prevent skew. The prediction cache short-circuits repeat inputs — enormously effective for recommendation and embedding workloads with Zipfian keys. Shadow and canary paths receive mirrored or weighted live traffic for candidate versions, with their outputs logged and judged offline (shadow) or guarded by metrics (canary). The model registry is the source of truth: immutable, signed artifacts with lineage — which data, which training run, which eval scores — and stage transitions (staging → production) that gate what the router may serve.
Bottom row — control. The autoscaler scales on concurrency and queue depth, not CPU (GPU-bound replicas can be saturated at 30% CPU), and respects the long replica warm-up: artifact pull, device memory load, JIT warm-up. The rollout controller automates the canary ramp — 1%, 5%, 25%, 100% — evaluating guardrail metrics at each step and reversing the weights automatically on breach.
End-to-end flow
Follow one ranking request through the stack. A client calls POST /v1/models/ranker:predict. The gateway authenticates the tenant, checks its token bucket, and stamps a deadline header: 80ms end-to-end. The router resolves ranker@prod: version 41 holds 95% of traffic, canary version 42 holds 5%; this request draws v41. In parallel, the handler probes the prediction cache (miss) and issues the feature fetch — user embedding, recent-activity aggregates — which returns in 6ms from the online store.
The request, now a tensor payload, enters v41's batching queue with 74ms of budget left. The batcher is running a 4ms max_queue_delay; within 2.1ms, seven other requests arrive, the bucket for this shape fills to max_batch_size=8, and the batch releases early. The GPU replica runs the forward pass in 11ms; per-request outputs are demultiplexed, and this caller gets its scores back through router and gateway at t=31ms — comfortably inside SLO, with the accelerator having amortized its 11ms across eight requests instead of one.
Meanwhile the same request was mirrored to the shadow deployment of version 43, an architecture change too risky even for canary. Shadow's response is not returned to the caller; it is logged next to v41's response and the request features, feeding an offline judge that compares the two distributions over millions of paired examples. Version 42's canary, by contrast, is live: the rollout controller is watching its p99, error rate, and a business guardrail (click-through delta) against v41's. At 2 a.m., v42's p99 breaches for ten consecutive minutes — a kernel regression under a rare input shape. The controller sets its weight to zero, pages nobody-critical, and the morning finds the ramp reversed with a diagnosis waiting in the paired logs.
Scale-out follows the same discipline. Evening traffic doubles queue depth; the autoscaler adds three replicas. Each pulls the signed v41 artifact from the registry, loads weights to device memory, runs warm-up batches until compile caches are hot, and only then reports ready — so the router never sends live traffic to a replica that would serve its first request at 30× normal latency.