A load test is an experiment, and most LLM load tests are badly designed experiments. The harness spins up N concurrent workers, each fires a prompt, waits for the full completion, and immediately fires another; the report says “we sustained X requests per second at p99 of Y.” That number is very often meaningless — not slightly off, but measuring something other than what production will do to you. Token streaming breaks the assumptions behind classic request/response load generation: a single request occupies the server for a variable and prompt-dependent duration, the server batches requests against each other, and the user-visible latency is a whole stream of events rather than one. This article is about generating and measuring load correctly, so that the ceiling you find is the real one.
What the test actually has to answer
Before choosing a tool, be precise about the question. A serving deployment has one interesting property: the relationship between offered load and user-visible latency. Everything else — a peak throughput figure, a concurrency setting, a GPU utilisation percentage — is a projection of that curve onto one axis, and projections lose the shape.
So the deliverable of a load test is a curve, not a number: for each level of offered load, the distribution of time to first token and of inter-token delay. From that curve you can read off whatever your service level requires; from a single “max QPS” you can read off nothing, because you cannot tell whether the system was comfortable at that point or already collapsing. The three stages below — generate, measure, ramp — each have a specific failure mode, and getting any one of them wrong invalidates the run.
Closed-loop generators cannot overload anything
The default shape of almost every load tool is closed-loop: a fixed population of virtual users, each of which sends a request, waits for the response, thinks for a moment, and sends the next. It feels realistic. It is also structurally incapable of overloading the system under test.
Little’s Law makes this exact. With N users, mean response time R and mean think time Z, the throughput is pinned at N / (R + Z). The clients are a negative feedback loop wrapped around the server: as the server slows, R grows, arrivals slow down, and the queue drains. There is no backlog because the load generator refuses to create one. You can raise N until latency is absurd, but you never observe the behaviour that matters in production — a queue growing without bound while traffic keeps arriving because real users do not politely wait their turn.
Model an arrival process instead
The fix is an open-loop generator: requests are issued on a schedule derived from a target arrival rate, independent of whether earlier requests have finished. If the target is 20 requests per second, the harness issues 20 per second whether the server is healthy or dying. Now overload is expressible, because arrivals and completions are decoupled.
Use a stochastic arrival process rather than a metronome. Real traffic arrives in clumps, and a Poisson process (exponentially distributed inter-arrival gaps) is the standard neutral model; it produces bursts that a uniform tick never generates, and bursts are where queueing latency is born. Most modern harnesses expose this directly — look for an executor or mode described as constant arrival rate, constant throughput, or open workload, and confirm it does not silently cap in-flight requests, because a cap quietly reintroduces closed-loop behaviour.
Coordinated omission, the tail you never see
Closed-loop harnesses do not merely fail to create overload; they actively hide it. When a request stalls, the worker thread that would have issued the next request is blocked waiting. Those suppressed requests would have landed during the stall and been slow too — but they were never sent, so they never appear in the sample. The measured tail is drawn disproportionately from the periods when the server was healthy. Gil Tene named this coordinated omission, and it routinely understates high percentiles by an order of magnitude.
The open-loop generator is the structural fix, but only if you record honestly. Timestamp each request at its intended dispatch time, not the moment the harness actually got around to sending it, and include requests that were rejected, timed out, or queued inside the client. A run whose error rows are dropped from the latency table is a run that reports its best behaviour only.
Prompt and output lengths are test parameters
For a conventional web service, request size is a detail. For an LLM service it is the dominant variable, because it determines how much of each request is prefill and how much is decode. Prefill processes the whole prompt in a compute-bound pass; decode emits one token at a time and is bound by reading weights and cache from memory. Your length distributions therefore choose the workload mix, and a test that fixes both lengths tests one point in a two-dimensional space.
Sample prompt lengths and output lengths from distributions taken from your own logs, and expect them to be skewed rather than normal — a long tail of large prompts, and output lengths shaped by whatever the application asks for. Two deployments with identical request rates and wildly different length profiles are different workloads. Record the realised distributions alongside the results; otherwise the run is not interpretable later.
Control the cache and the warm-up
A serving stack has state, and that state changes the answer. Cold-start effects include CUDA context creation, kernel autotuning and graph capture, memory-pool growth, and any compilation the runtime performs on first use. Measuring during that window measures the startup path, not steady state, so discard an explicit warm-up period rather than trusting an average to dilute it.
Prefix caching is the subtler trap. If every synthetic request shares the same system prompt or the same handful of prompt templates, the cache hit rate in the test can be far higher than in production, and prefill work simply disappears. Decide deliberately which regime you are measuring, then make the prompt corpus match it: a shared-prefix corpus if your product genuinely has one, otherwise distinct prompts with unique prefixes. Report the observed cache hit rate as a result, because it explains the numbers around it.
Record TTFT and inter-token latency as distributions
End-to-end request latency is nearly useless on its own for a streaming API: it conflates queueing, prefill, and a decode phase whose length is set by how many tokens the model happened to emit. A response that took eight seconds because it produced eight hundred tokens is not slow. Split the measurement to match what the user experiences.
Time to first token captures queueing plus prefill — the wait before anything appears. Inter-token latency, the gap between successive tokens, captures the streaming experience once it starts, and it matters as a distribution because a stream that averages a comfortable rate while stalling for a second mid-response reads as broken. Keep full histograms and report high percentiles; a mean over a batching scheduler hides exactly the stalls you are hunting. Normalising by output token count keeps runs with different length profiles comparable.
Find the knee, not the maximum
Ramp the arrival rate in steps, holding each step long enough to reach steady state, and plot latency percentiles against offered rate. The shape is a hockey stick: flat while the server absorbs arrivals, then a sharp upward turn once the arrival rate approaches the service rate and the queue stops draining between bursts. That turn is the knee, and it is the operational answer.
Peak throughput sits past the knee, in the region where the queue is growing and latency is climbing without bound; running there is how a service accumulates a backlog it never recovers from. Watch also for throughput that falls as load rises — a sign of retry storms, preemption, or admission thrashing. Two practical additions: run a step past the knee deliberately to confirm the system sheds load gracefully rather than falling over, and step back down to check it recovers.
Make the run reproducible
A load test earns its keep by being repeatable across code changes, so pin everything that moves. Fix the random seed for arrival gaps and length sampling, version the prompt corpus as a file rather than generating it inline, and record model, quantisation, runtime version, parallelism layout, batching and memory settings, and the exact hardware and driver stack.
Then check the harness itself is not the bottleneck. A client that parses server-sent events on one thread, or that runs a long way from the server, will attribute its own scheduling delay to the model. Verify the generator actually achieved the requested arrival rate — if realised rate lags target, the test silently became closed-loop and the results are void. Store raw per-request records, not summaries: percentiles cannot be recomputed or re-bucketed from a mean, and the question you want to ask next month is never the one you aggregated for.