Two teams can run the same model, the same inference engine and the same GPU SKU, and end up with cost-per-token an order of magnitude apart. The difference is almost never the kernel. It is the deployment topology: whether that GPU belongs to one tenant or many, whether it can fall to zero replicas between bursts, and whether you rent it by the hour or bought it outright. This piece walks the five topologies that cover almost all production LLM serving, asking each the same four questions — what isolation, what utilization, what cold-start behaviour, and which cost regime — then gives you a decision path from workload shape to topology.
Topology is three independent choices, not one
‘Deployment pattern’ sounds like a single pick from a menu. It is really three orthogonal decisions that people collapse into one word, which is why architecture arguments go in circles.
Tenancy: does one workload own the GPU, or do several share it? Elasticity: is the replica count fixed, autoscaled within a floor, or allowed to reach zero? Ownership: is the hardware owned capital, a multi-year reservation, on-demand rental, or preemptible spot? The named patterns are corners of that cube.
Separating the axes matters because they fail differently. Tenancy decisions get overturned by a compliance review, elasticity decisions by a cold start, ownership decisions by a demand forecast that was wrong. You can revisit one axis without touching the other two — but only if you did not fuse all three into one unnamed default.
The stack barely changes; the placement does
Topology is worth arguing about separately because the software underneath it is nearly constant. A scheduler, a paged KV cache, a batching loop, a weight-loading path, a metrics pipeline — the same components appear in all five topologies below, so treat the engine as a black box with a known cost curve. What the topology changes is everything around that box: who else is resident on the device, how long a new copy takes to exist, and who pays while it idles at 3am. Isolation, cold start, idle cost — those three are the comparison.
Dedicated single-tenant
One deployment, one model, GPUs reserved to it and nothing else. Assume this by default for a latency-critical, high-volume endpoint.
Isolation is total: no other tenant can evict your KV cache, steal SM time, or OOM your process. Your p99 is a function of your own traffic, which is the only way a tight tail-latency SLO is defensible. Utilization is the weak point — you provision for peak, so average utilization is roughly the inverse of your peak-to-mean ratio, and a diurnal consumer product with a 4:1 peak-to-trough easily parks below 30%. Cold start is a non-issue: replicas are always warm and scaling events are rare enough to afford being slow.
Cost regime: it wins when sustained demand is high enough that the GPUs would be busy anyway. Past the point where you would keep a replica warm around the clock regardless, dedicated is the cheapest form of that replica.
Pooled multi-tenant
Many models or many customers share one GPU fleet. This topology exists purely to attack the utilization number that dedicated cannot fix.
The economics are compelling because uncorrelated workloads smooth each other. Twenty tenants each with a 4:1 peak-to-mean ratio do not sum to a 4:1 fleet ratio — their peaks land at different times — so a pool runs at several times the average utilization of the same workloads deployed separately. On expensive accelerators that is the largest cost lever available.
The price is isolation, bought back with mechanisms rather than topology: hardware partitioning (MIG) gives each tenant a separate memory path and predictable throughput at the cost of stranded capacity, while process-level sharing (MPS) or time-slicing gives density with no fault isolation and a noisy-neighbour tail. Those trade-offs belong to the MIG and GPU-scheduling articles; the point here is that pooled means committing to one of them, plus admission control and per-tenant quota — otherwise your best customer's p99 becomes someone else's batch job.
Serverless and scale-to-zero
Scale-to-zero is what everyone wants for the long tail of models: pay only for the seconds you compute. For CPU workloads it is close to free. For LLMs it collides with physics.
A cold replica must schedule onto a GPU node, pull a container image that is often tens of gigabytes, fetch the weights, initialise the CUDA context, allocate the KV cache pool, and run warmup iterations to capture graphs and autotune kernels. The weight fetch dominates. Take a model whose weights are a few hundred gigabytes, pulled over a 100 GbE link at a theoretical 12.5 GB/s — and object stores rarely sustain line rate: that is tens of seconds at best, before any other step. Treat the figure as illustrative; the mechanism is the point. A first token a minute after the request arrives is not a serverless endpoint, it is a batch job with a synchronous API.
Cost regime: serverless wins decisively for spiky, low-duty-cycle, cold-start-tolerant traffic — internal tools, eval harnesses, per-customer fine-tunes queried a few times a day — and loses badly for anything interactive.
The cold-start ladder, and where it runs out
Because cold start is the gate on scale-to-zero, know the mitigations and where they stop. Cache weights on node-local NVMe so the next cold start reads locally; stream them layer-by-layer so early-layer compute overlaps later transfers; use a direct storage-to-GPU DMA path to skip the CPU bounce buffer; snapshot a warmed process and restore it; pre-pull images onto every GPU node. Each buys back real seconds.
But notice where they point. Node-local caches pin models to nodes, snapshot-restore holds memory, pre-pulled images mean the node is already up and billing. Push the ladder far enough and you have rebuilt a warm pool — you have converged on pooled multi-tenant with an aggressive scale-down floor, not on scale-to-zero. That is usually the right destination; arrive deliberately rather than after a quarter of latency incidents.
On-prem and owned fleets
Owning the hardware is a financing decision wearing an architecture costume. It turns on two things: the spend floor, and constraints that money cannot relax.
The spend-floor argument is arithmetic. Rental prices amortise the vendor's capital over an assumed utilization plus margin, so if your own sustained utilization would be high, you beat that. The break-even is not a headline hourly rate — it is that rate against purchase price over useful life, plus power, cooling, networking, spares and staff, all divided by the utilization you will genuinely achieve. Teams overestimate that last number badly; an owned cluster at 35% is usually dearer per token than reserved cloud capacity.
The constraint argument is simpler and stronger. Data residency, air-gapped environments, regulated workloads that cannot leave a facility, or data gravity heavy enough that egress dominates the bill — these pick the topology regardless of the arithmetic. When residency drives, design for the utilization you can reach inside the boundary.
Hybrid burst
Hybrid burst runs a steady base load on owned or reserved capacity and spills the peak to on-demand or spot cloud GPUs: the cheap floor of ownership with the elastic ceiling of rental.
In practice it is the hardest topology to operate, and the reasons are all about state. Weights must be resident, or fetchable fast, in both environments — storing hundreds of gigabytes twice and keeping versions in lockstep. The burst tier has a cold start exactly when you need it most, so the scale-out trigger must lead demand rather than follow it. Latency is asymmetric between tiers, so a request sent to the burst tier can silently miss an SLO the base tier meets. And because a conversation's KV cache lives where it started, sessions cannot be rebalanced mid-stream.
Cost regime: hybrid earns its complexity when the peak is both large and infrequent. Under roughly 2:1 peak-to-mean, buy the headroom and skip the second control plane.
The decision path from workload shape to topology
Work the constraints in order of how hard they are to negotiate: residency is a legal fact, the SLO is a product promise, traffic shape is measurable, and only the spend floor is genuinely a preference.
| Signal in the workload | What it forces |
|---|---|
| Data cannot leave a jurisdiction or facility | On-prem or a region inside the boundary — settled before any cost work |
| Interactive SLO, tight p99, steady volume | Dedicated single-tenant; scale-to-zero is off the table |
| Many small or per-customer models, low duty cycle | Pooled multi-tenant with adapter sharing, not one deployment each |
| Spiky, tolerates seconds of first-token delay | Serverless scale-to-zero |
| High sustained utilization, multi-year horizon | Owned or reserved capacity; on-demand is the expensive default |
| Large infrequent peak over a stable base | Hybrid burst, if a second control plane is affordable |
Where two rows conflict, the higher row wins. A residency constraint and a spiky profile do not average into a compromise — they mean running the spiky workload inside the boundary and accepting worse utilization.
Topology is a phase, not an identity
Each pattern is right for some period of a product's life and wrong later, and the transitions are predictable. A new endpoint starts serverless because traffic is unknown and idle GPUs are unaffordable. It graduates to a dedicated deployment when the SLO tightens or the cold start shows up in the p99. It merges into a pool once sibling models strand capacity in per-model deployments. It acquires reserved or owned capacity once demand is forecastable a year out.
What makes those moves cheap is clean seams: weights in a registry rather than baked into images, routing behind a stable gateway rather than client-side endpoints, per-model utilization and cost-per-token measured from day one so the next move is triggered by data. The teams that get stuck are the ones whose first topology leaked into their client contracts.