Picking an LLM serving engine is a procurement decision dressed up as an engineering one. The architecture of a serving stack is largely settled — a scheduler, a KV-cache manager, a model executor, a streaming layer — and the major open engines all implement that same shape. What differs is which problem each one was built to solve first, and that shows up in the parts you will actually fight with in production: admission and preemption, cache-block reuse, which quantized formats load, whether grammar-constrained output is first-class, how many adapters and modalities one replica holds, and how legible the thing is at 3 a.m. This piece gives you those axes and a way to weight them against your workload — not a leaderboard.
Why this is not a benchmark question
The instinct is to look for a throughput chart and pick the tallest bar. Resist it. Serving-engine benchmarks are extremely sensitive to their conditions: prompt and generation length distribution, arrival rate, admission policy, quantization format, tensor-parallel degree, GPU generation, driver version, and whether the harness measured time-to-first-token, inter-token latency, or aggregate tokens per second. Change one and the ordering moves.
Worse, these projects iterate on a scale of weeks. A kernel that was missing in one release is merged in the next; a format that was unsupported ships behind a flag; an API is renamed. Any feature matrix you read — including the framing in this article — is a snapshot, and the specifics must be re-checked against the exact release you intend to deploy. What is durable is the set of axes: the questions worth asking survive far longer than any particular answer. Evaluate on your own traffic shape (see gpu_llm_load_test); treat published numbers as a hint, never a decision.
Four engines and their design centers
Rather than a capability grid, hold each project's original problem in mind — it predicts where the sharp edges are.
vLLM came out of the paged-KV-cache line of work and is built as a broad, open, model-agnostic serving engine: get many architectures running well on commodity accelerators with good memory utilisation. TensorRT-LLM comes from the compiler side — you build an optimised engine ahead of time against NVIDIA hardware, trading flexibility and a build step for tight control of the generated kernels. TGI (Text Generation Inference) grew out of running a very large public model catalogue as a service, so its centre of gravity is Hub integration and operational packaging. SGLang started from the language and program side — structured generation, prefix reuse across calls, multi-step programs — and its runtime reflects that.
Axis 1 — the scheduling model
Every serious engine now schedules at the granularity of a decode step rather than a whole request, so the presence of continuous batching is not a differentiator; the policy around it is. Ask what the admission rule is when the cache is nearly full, what happens to an in-flight request when a larger one needs blocks (recompute versus swap versus reject), whether prefill and decode contend in the same step or are chunked, and whether you can express priority classes or per-tenant fairness at all.
This axis dominates tail latency. An engine that preempts by recomputing a long prompt will produce a rare but brutal p99 spike; one that caps prefill work per step protects interactive users at some cost to bulk throughput. If you run mixed traffic — chat alongside long-document summarisation — scheduling policy is probably your highest-weight axis, because it decides whether the batch job can stall the chat. The mechanism itself is covered in gpu_continuous_batching.
Axis 2 — KV-cache management
Cache capacity, not compute, is usually what caps concurrency, so how an engine manages it is the second thing to interrogate. Block-based paged caches are near-universal now, but the surrounding machinery varies: whether identical prefixes are shared automatically or need an explicit hint, whether a prefix cache survives across requests and for how long, whether blocks can spill to host memory or NVMe, and how much of the free-block accounting you can see.
Weight this axis heavily when your prompts share structure. A long system prompt, a retrieved-document preamble, or a multi-turn conversation that replays history means most of your prefill is redundant work that an engine with strong automatic prefix reuse simply skips. Conversely, if every request is a short, unique prompt, prefix sharing buys nothing and you should weight raw block efficiency and eviction behaviour instead. The mechanism lives in gpu_paged_kv_cache.
Axis 3 — quantization support
Quantization is where engine choice most often becomes a hard constraint, because support is not a single yes/no. Separate three questions. Which formats can be loaded — weight-only integer schemes, weight-and-activation low precision, KV-cache quantization? Which of those have fused kernels for your GPU generation, rather than a slow dequantize-then-GEMM fallback? And can the engine consume checkpoints someone else already produced, or does it require its own conversion or calibration pass?
That last question is the operational one. A conversion step means a build pipeline, artifacts to store and version, and a re-run every time you change models — real cost, sometimes worth paying. Two traps to avoid: assuming a format name implies the same numerics everywhere, and assuming a quantized checkpoint published for one engine loads unchanged into another. Verify both with the actual model you plan to serve, and re-check quality, not just that it loads.
Axis 4 — structured output and grammar support
If downstream code parses the model's output, constrained decoding stops being a nicety. The relevant questions: can the engine mask logits against a JSON Schema, a regular expression, or a full context-free grammar? Is that implemented inside the engine, where the mask can be computed alongside the decode step, or bolted on outside it? Is the grammar compiled once and cached, or recompiled per request? And — the one people forget — does it still work combined with the other features you need, such as speculative decoding or a quantized model?
The cost profile matters as much as the capability. A poorly integrated constraint layer serialises the batch: every step waits on a per-request mask computed on the host. If you are building an agent or a tool-calling API where essentially every response must validate, weight this axis near the top; if you serve free-form chat, weight it near zero.
Axis 5 — LoRA multiplexing and multi-modal inputs
These are two questions about the same thing: how much variety can one replica absorb before you need another fleet? For LoRA, ask whether many adapters can be served concurrently against one base model in a single batch, whether adapters can be loaded and evicted at runtime without a restart, what the per-adapter memory cost is, and what happens to throughput as the number of distinct adapters in a batch grows. The alternative — one deployment per fine-tune — is a fleet-sizing disaster once you have dozens of tenants.
For multi-modal, the questions are structural: is the vision or audio encoder run inside the same process and scheduled with the decode loop, or is it a separate service you must operate? How are image tokens accounted for in the cache budget? Are the specific model families you care about actually wired up, given that multi-modal support is nearly always per-architecture rather than generic?
Axis 6 — the distributed and multi-node story
Single-GPU behaviour tells you little about how an engine behaves when the model does not fit. Establish which parallelism modes exist — tensor, pipeline, expert — and, more importantly, which combinations are actually exercised. Ask how workers are launched and discovered, whether the engine assumes a particular launcher or cluster runtime, how it behaves when one rank dies (does the replica fail fast and get rescheduled, or hang holding memory?), and whether prefill and decode can be placed on separate hardware.
Also check fabric assumptions: an engine tuned for a single NVLink-connected node may degrade when tensor parallelism crosses a slower interconnect — a topology question more than an engine question (see gpu_nvlink, gpu_infiniband). If your models fit on one node, this axis weights to zero, which is itself an argument for choosing models that fit.
Axis 7 — observability and operational maturity
This is the axis teams underweight and then regret. On observability: are there first-class metrics for the numbers that matter — queue depth, running versus waiting requests, KV-cache utilisation, preemption count, time-to-first-token and inter-token latency distributions — or only process-level counters? Is there a health endpoint that reflects scheduler state rather than merely that the process is alive? Can you trace a single request through admission, prefill and decode?
On maturity, the questions are social. How often do releases land, and how often do they break configuration or API surface? Is there a stability policy, or does every upgrade need a full requalification? Is support a vendor contract or a community issue tracker — and is that acceptable to whoever carries the pager? A fast-moving project can be the right call, but only if you have pinned versions, a canary path (gpu_llm_canary) and someone who can read the source when it breaks.
A decision framework — matching axis to workload
Weight the axes by workload rather than scoring engines in the abstract:
| If your workload is… | Weight most heavily |
|---|---|
| Interactive chat with mixed request sizes | Scheduling policy and preemption behaviour |
| Long shared system prompts or RAG preambles | Prefix reuse and cache lifetime |
| Agents and tool calling | Structured output integration and its cost |
| Many tenants on one base model | LoRA multiplexing and adapter hot-swap |
| One model, fixed shape, cost-driven | Quantization formats and kernel coverage |
| Models larger than one node | Parallelism combinations and failure handling |
| A small team carrying the pager | Observability surface and upgrade cadence |
Then run the only test that settles it: replay a sample of your own traffic through two or three candidates on the hardware you will actually buy, measure the percentiles you are held to, and count the days each took to get there. That integration cost is a real result, not overhead.