Evidence label: deterministic synthetic model. This page runs a small JavaScript model of an autoscaled LLM fleet. Synthetic requests arrive at a time-varying rate, each with a random output length, and are routed to the least-loaded ready replica, which decodes all its sequences together using a linear step-time model. An HPA-style controller evaluates the chosen signal every 15 seconds and adds or drains replicas, and new replicas serve only after the cold start. Nothing here runs a model, a GPU or Kubernetes; every number is a calculated estimate.
Play the lab
How to play
Choose a scaling signal and a target in that signal's units: percent for GPU utilization, requests per second, waiting requests, or running plus waiting requests per replica. Then set base traffic, the minimum replica count, the cold start and the scale-down window. Every change replays the same twenty minutes of traffic with the same seed, so differences come from your settings. The cost tile prices the replicas that existed (including those starting and draining) at an illustrative $4 per GPU-hour.
Missions unlock in order and award up to three stars; most reward a lower bill once the latency goal is met. Toggle the events to stress your configuration, and use a new seed to check that a fix is not luck.
What the model assumes
Base traffic ramps from 60% to 100% of the chosen rate over ten minutes. The launch spike multiplies it by 2.5 between 480 and 800 seconds with 20-second ramps. Outputs are exponential with a mean of 240 tokens, rising to 720 between 300 and 540 seconds with the longer-answers event. Each replica holds up to 64 sequences and a decode step takes 6 ms plus 0.25 ms per sequence; prefill is ignored. GPU utilization is reported as 93% to 100% whenever a replica has any work. The controller averages over 30 seconds, uses a 10% tolerance, scales up immediately, and scales down at most one replica per evaluation after the stabilization window. A request meets the SLO if its first token arrives within 2 seconds.
What it cannot tell you
Real replicas differ in prefill cost, KV cache limits, preemption and prompt caching; node provisioning, GPU availability and image caching make cold starts far more variable than one slider; and real routers, gateways and retries add their own queues. The model also uses a single traffic class. Use it to understand why signals behave differently, then validate your own autoscaler with a ramped load test on real hardware.
Read the full explanation
- Autoscaling LLM Servers: Why GPU Utilization Lies and Queue Depth Wins (companion article)
- TTFT, TPOT and Goodput: Setting LLM Latency SLOs That Mean Something
- Continuous Batching Explained: Why Static Batches Leave GPUs Idle
All AI in Production labs · Lab catalog
- Kubernetes documentation: Horizontal Pod Autoscaling
- KEDA documentation: Prometheus scaler
- NVIDIA NVML API reference: nvmlUtilization_t
- Google Kubernetes Engine: best practices for autoscaling LLM inference workloads with GPUs
- Knative documentation: configuring concurrency
Views are my own and do not represent any employer, past or present.