Why architecture matters here
Architecture matters here because the naive configuration fails in both directions at once, expensively. Scale on GPU utilisation with a 70% target and a continuous-batching server will sit at 95% utilisation while serving four requests comfortably, so you scale up into an empty room. Then a burst arrives, utilisation is already pinned, the signal does not move, and you do not scale when you need to. You have built a controller that adds capacity when it is not needed and withholds it when it is — worse than no autoscaling, because it costs more and hides the problem.
The economics amplify every mistake. A CPU replica costs pennies an hour, so over-provisioning by 30% is a rounding error nobody audits. An accelerator replica can cost several dollars an hour, so the same 30% is a line item that gets a meeting. This inverts the usual bias: for web services the default is 'over-provision, it's cheap'; for inference the default pressure is to run lean, which means you meet the cold-start ladder constantly rather than rarely.
Capacity is also not guaranteed to exist. Autoscaling policies implicitly assume that asking for a node produces a node. In accelerator-constrained regions that assumption fails, and it fails exactly during the correlated demand spikes when you most need it. A design that has no answer for 'the scale-up request was refused' will simply queue until it times out. The answer has to be architectural — reservations, multi-region fallback, a smaller fallback model, or explicit shedding — because it cannot be a retry.
Finally, the workload itself is not uniform. A request with a 200-token prompt and a 20-token response and one with a 60,000-token prompt and a 4,000-token response are both 'one request', and they differ in cost by orders of magnitude. Any controller counting requests per second is measuring a number whose relationship to load changes with traffic mix. Scaling on a metric that already integrates the work — queue depth in tokens, or admitted batch occupancy — is what makes the loop stable when the mix shifts.
The architecture: every piece explained
The admission queue is the load-bearing component, and it is deliberate rather than incidental. Requests arrive, are enqueued, and wait for a slot in the running batch. Its depth is the cleanest available statement of supply versus demand: zero means you have spare capacity, growing means arrivals exceed service rate, and the rate of growth tells you how badly. Unlike utilisation it is unambiguous, and unlike request rate it reflects actual work because a heavy request occupies its slot longer.
The metric exporter publishes queue depth alongside TTFT percentiles and batch occupancy. TTFT is the user-facing translation of queue depth and belongs in the SLO; queue depth belongs in the controller because it moves first. Exporting both, and understanding that one leads the other, is what lets you set a scaling target that means something to the business: 'scale so that p95 TTFT stays under 800ms' becomes 'hold mean queue depth near two' once you have measured the relationship on your own hardware.
The scaling controller runs target-tracking with asymmetric hysteresis. It compares mean queue depth to a target and computes a desired replica count, but the up and down paths are deliberately different: scale up fast on a short window because being late is a user-visible outage, and scale down slowly on a long window because being early means paying the cold-start ladder again in ninety seconds. Symmetric hysteresis produces thrash, and thrash on a four-minute actuator is not oscillation you can ignore — it is a permanent state of having the wrong capacity.
The provisioner and cold-start ladder are where the minutes live, and decomposing them is the highest-leverage work available. The rungs are node acquisition, container image pull, weight download and load into device memory, kernel compilation or graph capture, and warmup inference. Each has a different fix: node acquisition is answered by warm pools or reservations, image pull by slimmer images and pre-pulled caches, weight loading by local NVMe caches and memory-mapped formats, compilation by shipping precompiled artifacts, and warmup by a synthetic request at startup. Teams that measure the ladder usually find one rung dominating and fix it in a week; teams that treat 'cold start is four minutes' as atomic never fix anything.
The warm pool, load shedder, and cost governor handle the residue. The warm pool holds replicas past the ladder but idle — the premium you pay to make some fraction of scale-ups instant. The load shedder covers the window where arrivals exceed capacity and scaling has not landed yet, rejecting or degrading at the queue's edge with a clear error rather than letting depth grow until everything times out. The cost governor caps the whole loop: a maximum replica count, a spot and on-demand mix, and a budget ceiling that turns a retry storm from a five-figure surprise into a shed-load incident.
End-to-end flow
Trace a burst. It is 09:00 and a customer's scheduled batch job begins; arrivals jump from 20 to 200 per second in under ten seconds. The eight running replicas are batching happily at depth zero. Within two seconds the queue is at 40 and climbing, and TTFT p95 has gone from 400ms to 1.9s. Nothing is broken yet — the queue is doing exactly its job, absorbing the mismatch and converting it into latency instead of errors.
The controller's short up-window elapses at four seconds. Mean depth of 40 against a target of two yields a desired count far above eight; the controller clamps to its per-step scale-up limit and requests twelve replicas. Four come from the warm pool and are serving within fifteen seconds — that is the insurance premium paying out, and it is the difference between a blip and an incident. The other eight enter the cold-start ladder: node acquisition, image pull from the regional cache, weights from local NVMe, precompiled kernels, one synthetic warmup request.
At thirty seconds the warm-pool replicas have pulled depth down to 25, but arrivals are still 200/s and the ladder has three minutes to run. Depth crosses the shed threshold. The shedder begins rejecting the lowest-priority tier — the batch job's own requests, which carry a retryable class — with a 429 and a Retry-After. Interactive traffic keeps flowing at a degraded but honest TTFT. This is the design working as intended: the burst's originator absorbs the burst's cost, and the shedding is a deliberate choice rather than a timeout cascade.
At 3m40s the cold replicas finish warmup and join the pool. Depth falls through the shed threshold, shedding stops, and TTFT returns under SLO. The cost governor notes that twenty replicas is within the cap and does nothing. Had the burst been three times larger, the governor would have clamped at the cap and the shedder would have carried the remainder indefinitely — which is the correct behaviour, because the alternative is an unbounded bill driven by whoever can generate the most traffic.
At 09:20 the batch job completes and arrivals drop to 20/s. Depth is zero. The controller computes a desired count of eight, but the long down-window means nothing happens for ten minutes — deliberately, because a second batch job at 09:25 is likelier than the traffic staying flat, and holding twelve idle replicas for ten minutes is cheaper than climbing the ladder again. When the window finally expires with depth still zero, replicas retire a few at a time, warm pool refills to its floor, and the system returns to its resting premium.
Step back and count where the minutes actually went, because that accounting is the design's whole justification. The burst arrived over ten seconds; the controller reacted in four; the warm-pool replicas landed in fifteen; the cold replicas took nearly four minutes. For that entire four-minute window the loop could not add real capacity no matter how aggressively it tried, and everything that kept the endpoint honest during it — the queue absorbing the mismatch as latency, the warm pool paying out instantly, the shedder protecting interactive traffic, the cost governor bounding the blast radius — was a decision made before the burst, not a reaction to it. A team that had tuned only the controller's gains and left the warm pool at zero would have had an identical four-minute hole with nothing in it, and the incident review would have concluded, wrongly, that the autoscaler was too slow. The autoscaler was exactly as slow as physics; the difference between a blip and an outage was how much latency insurance had been prepaid and how gracefully the system degraded while the insurance that had not been bought was being manufactured on the cold-start ladder.