Why architecture matters here
Architecture matters for LLM serving in a way it did not matter for classic web services. A REST endpoint that returns JSON in 20 milliseconds has one bottleneck — the database — and one scaling axis: horizontal replicas. An LLM endpoint that returns 1,000 tokens has at least six bottlenecks (compute, HBM bandwidth, interconnect bandwidth, KV cache memory, tokenizer speed, and network egress) and three scaling axes (data parallel, tensor parallel, and pipeline parallel). The wrong architecture wastes a $30,000 GPU sitting mostly idle.
Latency budgets tell you why the pipeline shape matters. A conversational assistant targets 300 milliseconds to first token and 30 tokens per second thereafter. A summarization batch job tolerates 30 seconds per document but wants maximum throughput. An agentic loop makes ten LLM calls per action and cares about consistent tail latency. Each of these workloads maps to a different admission policy, batching strategy, and even model choice — but they share the same underlying pipeline shape.
Cost matters just as much. On modern H100 SXM nodes the marginal cost of one token can vary by a factor of ten depending on how well the stack keeps GPUs full. Continuous batching (Orca, 2022) alone gave a 2-5x throughput lift over static batching. Paged attention (vLLM, 2023) gave another 2-4x by eliminating fragmentation. The prefill/decode split, expert parallelism, speculative decoding, and quantization each contribute another meaningful step. Miss two of these and your cost per token doubles. Understanding the architecture is how you notice you are missing them.
The architecture: every box explained
The diagram above compresses a lot of engineering into eight boxes. Read it left-to-right, top-to-bottom, following the arrows the way a real request flows.
Client / SDK. The entry point is a browser chat UI, an agent runtime, or a batch job. It sends a JSON payload — messages, model, temperature, max tokens — over HTTPS. The client is responsible for streaming decoded tokens back to the user via Server-Sent Events (SSE) or a gRPC stream. The SDK handles retries and, ideally, request-scoped tracing headers so that a slow response can be attributed to the correct backend hop later.
API Gateway. The gateway terminates TLS, authenticates the JWT, checks rate limits, enforces tenant quotas, and logs the request for audit. It knows nothing about tokens or attention — it is a policy layer. Common implementations wrap Envoy, Kong, or a cloud provider gateway. Gateways add 5-20 ms of latency but are essential for multi-tenant serving because they enforce the fairness contract that keeps one aggressive tenant from starving everyone else.
Router / Load Balancer. The router selects a backend region, a tier (free vs paid), and a specific model version. Sticky routing for conversation continuity is optional. Modern LLM routers also inspect the request to decide whether it should be handled by a small fast model or a large slow one; that "prompt routing" step is worth 30-50% cost reduction if done well.
Admission Control. This is where the pipeline first says no. Based on current queue depth, projected SLO burn, and tenant priority, admission control either accepts, defers, or rejects the request with a 429 and a Retry-After hint. Little's law is the underlying math: L = λ · W. If arrivals λ exceed service rate μ, queue length L grows without bound; you must reject to stay stable. Well-tuned admission control prevents cascading failure during traffic spikes.
Iteration Scheduler. This is the heart of modern LLM serving. Instead of batching whole requests, the scheduler makes decisions every token step: which requests to admit into the current batch, which to preempt, how to interleave prefill (compute-bound) with decode (memory-bound) work. Continuous batching, chunked prefill, and priority scheduling all live here. vLLM, TGI, and TRT-LLM each ship a scheduler you can study; the algorithms differ but the shape is the same.
Prefill Worker. Prefill runs one forward pass over the entire prompt. Attention is O(n²·d) in prompt length, so time is roughly linear in tokens for compute-bound configs. Flash attention keeps memory linear rather than quadratic. Prefill produces both the first output token and the initial KV cache entries.
Decode Worker. Decode generates one token at a time, reading the entire KV cache each step. Each step is memory-bandwidth bound because it reads a lot of state to produce a little state. Batching multiple decode requests together hides latency; that batching is why continuous scheduling exists.
Paged KV Cache. Instead of allocating a contiguous KV region per request, paged attention allocates fixed-size blocks and maps them via a block table. Fragmentation drops from ~50% to under 5%. Prefix sharing lets two requests with the same system prompt reuse the same physical blocks, saving huge amounts of HBM in chat-style workloads.
Model Weights and Fabric. The last row is the raw hardware substrate: model weights live in HBM, sharded across GPUs via tensor and pipeline parallelism, and the fabric — NVLink within a node, InfiniBand or RoCE between nodes — carries the collective operations (all-reduce, all-gather) that stitch the shards back together. If your model is 70 billion parameters at FP16, that is 140 GB of weights; you cannot fit it on one 80 GB H100, so this row is where "sharding" turns from a diagram detail into an operational reality.
End-to-end request flow
Trace a request through the stack. The client posts a chat completion request to the gateway. TLS terminates, the JWT is verified, and the request is annotated with a tenant ID and a tier. The router picks a region — the one closest to the user, with capacity, running the requested model — and forwards the request.
Admission control examines the current backlog. If p99 latency SLO has 40% budget remaining, the request is admitted. It is enqueued for the iteration scheduler with priority derived from the tenant tier and the request deadline.
On the next iteration, the scheduler looks at the mix of pending prefills and ongoing decodes. It grabs the request from the queue, splits its 4,000-token prompt into 512-token prefill chunks, and interleaves those chunks with the ongoing decode batch. This is "chunked prefill," the Sarathi-Serve trick that keeps a large prompt from monopolizing GPU time and starving concurrent decodes.
The prefill chunks flow through the model. Attention reads tokens from the prompt, produces K and V tensors, and writes them into paged blocks in the KV cache. The block table records where each block lives so the decode step can find them. The last chunk produces the first output token and the request transitions from prefill to decode.
Each subsequent decode step reads the entire cached K/V state for this request, plus intermingled other requests, produces one new token, and appends new K/V entries. The output token is streamed back through the gateway to the client as an SSE event within ~30 ms of generation. This continues until the model emits an end-of-turn token or the max-tokens limit is hit.
All of this coordination — thousands of requests, tens of thousands of forward passes, terabytes of KV data movement — happens per GPU per second. NCCL orchestrates the collectives that hold the tensor-parallel shards in sync. When it works, it is invisible. When it does not, everything gets slow at once.