Why architecture matters here
A GPU serving stack is the most expensive infrastructure most companies run per unit of throughput. An 8x H100 SXM node lists at roughly $250,000 to buy and $30 to $50 per hour to rent. If your stack keeps those GPUs at 50% utilization instead of 80%, you are paying 60% more per token than a well-tuned competitor. Architecture choices — pod shape, sharding, batching, autoscaling — are the difference between profitable serving and money-losing serving.
Latency requirements shape the stack. Interactive chat needs 300 ms to first token, which forces small models, aggressive batching, and low-overhead scheduling. Long-context summarization tolerates seconds and rewards larger models with high throughput. Agentic workloads have bursty patterns and small per-call payloads, which stress admission control and cold-start behavior. The right architecture is the one that matches the workload, not the one from the paper.
Reliability is the last driver. GPUs fail. NICs fail. Kubernetes upgrades roll pods you did not expect to roll. A stack that treats these events as routine — with rolling updates, graceful drain, and rapid failover — is one that customers do not notice. A stack that treats them as incidents is one that gets a two-hour outage per month. Architecture shapes which kind of team you become.
The architecture: every box explained
The diagram is a Kubernetes-native serving stack; each layer maps to a distinct concern, and each has open-source and commercial choices. Read it top to bottom.
L7 Ingress. This is where user traffic enters your cluster. TLS terminates here. HTTP/2 or gRPC frames get parsed. Ingress controllers (nginx, HAProxy, Envoy, or cloud LB) also handle traffic splitting for canary rollouts. The ingress is a global concern: one misconfiguration takes down every downstream service. Keep the config in git, deploy with the same rigor as your model, and always leave headroom for a burst.
Service Mesh. Inside the cluster, Istio, Linkerd, or Consul provides mTLS, retries, and circuit-breaking between services. For LLM serving the mesh matters because it enforces a consistent retry policy across all downstream consumers. Without it, one team's aggressive retry storm can crush a shared LLM backend. With it, you have one place to change policy and enforce backoff.
Model Router. A separate service that inspects the request and picks the right model backend. Simple routers use tenant tier. Sophisticated routers classify the prompt, decide whether it needs a big model or a small one, and route accordingly. The best router in production today is the combination of a small BERT-style classifier plus a rules layer for high-value routes; this pattern reduces GPU cost by 30-50% for many workloads.
Serving Engine. vLLM, TensorRT-LLM, TGI, or SGLang. This is the process that owns the GPU. It manages the scheduler, the paged KV cache, the CUDA graphs, and the batching. Every optimization you have read about — continuous batching, chunked prefill, speculative decoding — lives here. Pick one, understand it deeply, and stay on the release cadence. The engines evolve monthly.
Autoscaler. Horizontal Pod Autoscaler is the baseline; queue-aware scalers (KEDA, custom) work better for LLM workloads because they scale on request queue depth rather than CPU utilization. GPU pods take minutes to become ready — you must scale ahead of demand, not chase it. Reserve capacity buffers for burst; use spot GPUs only where restart is cheap.
GPU Pod. The pod is a full 8-GPU node with NVSwitch giving 900 GB/s NVLink between every pair. Tensor parallelism (TP=8) splits the model across GPUs; the fabric handles the all-reduce that reassembles the results after each layer. The pod also owns the paged KV cache, the CUDA graph compilation cache, and the pre-loaded model weights.
InfiniBand NDR. Between pods, InfiniBand NDR at 400 Gbps carries the collectives that stitch pipeline-parallel or data-parallel replicas together. RDMA lets the NIC talk directly to GPU memory via GPUDirect, bypassing the CPU. If your inference is single-pod, this layer is optional; if it is multi-pod, it is the whole game.
Object Store. S3, GCS, or an in-house object store holds model weights, tokenizer files, and adapter checkpoints. Weights load once at pod startup — often 100+ GB per pod — so the object store's throughput and locality matter for cold-start time. Region-local buckets and pre-warmed caches cut cold-start from tens of minutes to a couple of minutes.
Observability. Every layer emits metrics, traces, and logs. OpenTelemetry is the standard collector; Prometheus + Grafana handle metrics; Loki, Elastic, or a commercial tool handles logs; Jaeger or Tempo handles traces. The observability layer is what turns "the LLM is slow" into "the KV cache is under pressure in pod 47." Invest here early.
End-to-end pod lifecycle
Boot a fresh pod and walk through it. The Kubernetes scheduler places the pod on a node that has 8x H100 SXM devices free and matches the resource requests. The init container pulls the model weights from the object store — 140 GB for a 70B FP16 model — decrypts them if encrypted, and materializes them into a shared volume. The main container starts vLLM, which mmap-loads the weights, launches worker processes per GPU, and initializes the paged KV cache pool sized to the remaining HBM.
The pod registers with the service mesh via a sidecar. It publishes readiness (weights loaded, KV pool sized) and health (all workers responsive) endpoints. The autoscaler and mesh discover the new pod. Once readiness passes, traffic begins to flow to it via the router.
A request arrives. The ingress terminates TLS. The mesh checks mTLS. The router picks this pod based on model, tier, and current load. The serving engine's scheduler admits the request into the next iteration. Prefill runs; decode runs; the KV cache fills; tokens stream back. Metrics tag each step with a trace ID so you can reconstruct latency after the fact.
When traffic falls, the autoscaler scales the pod down after a cool-down period. The pod drains gracefully — it stops accepting new requests, waits for in-flight requests to finish, unregisters from the mesh, and terminates. When traffic rises, a new pod boots. The cycle repeats.
The steady-state cost per token is a function of how often that pod is in warm serving state versus cold or draining. Autoscaler tuning and pod pre-warming shape this ratio. Teams that get it right run 30-50% cheaper than teams that treat it as an afterthought.