Why architecture matters here
The single most consequential choice in load testing is invisible in most tools' quickstarts: the open versus closed workload model. A closed-loop generator runs N virtual users who each send a request, wait for the response, then send the next. When the target slows down, the generator slows down with it — offered load falls exactly when stress rises, which is precisely how real traffic does not behave. Real users and upstream services keep arriving regardless of your latency; queues build. An open-model generator fires requests on a schedule (a Poisson or measured arrival process) whether or not previous responses returned, which is why only open models reproduce the hockey-stick collapse where queueing delay explodes past the saturation point. Closed-loop tests routinely certify systems for 3x the load that later melts them.
The sibling trap is coordinated omission, named by Gil Tene: when a generator blocks on a slow response, it also fails to measure the requests it would have sent during the stall, so the recorded histogram is missing exactly the samples from the worst moments. P99 can under-report by orders of magnitude. Correcting for it — intended-start-time accounting, as wrk2 and properly configured k6/Gatling do — is an architectural property of the harness, not a report option you toggle later.
Architecture matters, finally, because the result must be trustworthy enough to gate decisions. A test whose generators saturated their own CPUs, whose data reused ten hot users against a cache, or whose environment shared a database with staging produces a number indistinguishable from a real one until production disagrees. Every component in the harness exists to defend the validity of the verdict — that is the lens for the rest of this article.
The architecture: every piece explained
Workload model and scenario definition. The model encodes what "realistic" means: the operation mix (reads to writes, endpoint distribution), arrival-rate curves by time of day, payload size distributions, and session shapes — derived from production traffic capture, not intuition. Scenarios are code, versioned with the services they test, so a test failure bisects like any regression. The model is where realism lives; everything downstream just executes it faithfully.
Controller and generator fleet. The controller compiles the ramp plan — warm-up, baseline, step or linear ramp, soak, spike — and distributes work to a fleet of generator workers (k6, Gatling, Locust workers, or purpose-built clients). Workers implement the open model: schedule arrivals, record every request against its intended start time into HDR histograms, and stream results without buffering surprises. The fleet scales horizontally precisely so no worker runs hot: a generator above roughly 70% CPU is manufacturing latency and attributing it to the target.
Test data pool and environment. The pool provides unique, realistic entities — users, sessions, tokens, carts — at production cardinality, because ten looping users test the cache, not the database. Data is partitioned across workers to avoid accidental contention on the same rows. The target environment's job is to be prod-shaped where it matters: same instance types and limits, same data volumes, same dependency topology (with third parties stubbed at realistic latencies, not zero). Every divergence is a documented caveat on the verdict.
Telemetry pipeline and analysis. Client-side histograms alone cannot distinguish target saturation from network artifacts; server-side metrics alone miss what users experienced. The pipeline merges both into one timeline — client percentiles over server queue depth, GC pauses, connection counts, CPU — keyed by test run. The results store keeps baselines per scenario; analysis compares runs against SLO thresholds and historical envelopes, flagging regressions statistically rather than by eyeball. The verdict feeds two consumers: release gates (pass/fail) and capacity planning (the measured knee of the latency-throughput curve, plus headroom against forecast).
End-to-end flow
A well-run test is a pipeline with a beginning in production and an end in a decision. It starts with traffic archaeology: sample a representative window of prod traffic, extract the operation mix, arrival curves, and payload distributions, and encode them as the workload model. Define the questions before the ramp plan — "does checkout hold P99 < 400ms at 2x Black Friday?" is testable; "see how it does" is not.
Preparation is unglamorous and decisive: seed the data pool to production cardinality, verify the environment's parity checklist (instance sizes, autoscaler settings, feature flags, dependency stubs at measured latencies), and — the step most teams skip — calibrate the harness against a known target: run the fleet against a static endpoint to confirm the generators can produce the planned peak with flat latency and idle CPU. Now the harness is an instrument with a zero point.
Execution follows the ramp plan. Warm-up brings JITs, caches, and autoscalers to steady state, excluded from results. Baseline establishes the healthy signature at moderate load. The ramp climbs in steps long enough for queues to reach equilibrium — five-minute steps are a reasonable floor — while the merged timeline is watched live: client P99 against server queue depth and saturation metrics. The test continues past the SLO breach to find the actual knee and the failure mode beyond it (graceful shedding or collapse?), because the shape of degradation is a finding, not noise. A soak phase at expected peak then runs for hours to surface leaks, fragmentation, and connection churn that step tests cannot see.
Analysis closes the loop: percentile curves versus baseline, knee location versus forecast traffic, headroom arithmetic, and a written verdict with caveats (what diverged from prod, what was stubbed). The scenario and thresholds then enter CI as a nightly regression gate at sub-peak load — cheap, continuous, and the reason the next big test holds no surprises.