Why architecture matters here
The architecture matters because concurrency is the single lever that most directly sets both the latency and the cost of a Cloud Run service, and it is the one most teams leave at a default they never examined. A service that is almost entirely I/O-bound — it receives a request, calls a database, calls another API, assembles a response — can serve dozens or hundreds of requests per instance profitably, because each request's CPU footprint is tiny and its wall-clock time is dominated by waiting. Set concurrency high and you need a fraction of the instances, which is a fraction of the bill. Set it to one and you pay for a whole idle container per in-flight request.
The inverse is equally important. A service that is CPU-bound — it does image processing, cryptography, template rendering, or model inference on the CPU — has no spare cycles to hand a second request while the first is running. Packing more concurrent requests onto such an instance does not increase throughput; it increases per-request latency, because now several requests are time-slicing a saturated CPU. For those workloads a low concurrency, sometimes one, is correct, and the cost of extra instances is the price of not degrading everyone's latency. The architecture forces you to know which kind of workload you have, because the same knob helps one and hurts the other.
It matters for correctness, too, in a way single-request platforms let developers ignore. When one container handles many requests at once, the handler runs concurrently against shared process state — module-level variables, connection pools, in-memory caches. Code that was safe when each request had its own process can now race: two requests mutate the same global, or a non-thread-safe client is used from multiple requests simultaneously. Moving from concurrency one to concurrency many is not a config change you make blindly; it is a statement that your handler is safe to run in parallel with itself.
It matters for the health of everything downstream. Concurrency multiplied by instance count is, roughly, the maximum number of simultaneous requests your service can send to its own dependencies. A service that scales to hundreds of instances each running high concurrency can open thousands of simultaneous connections to a database that tops out at a few hundred — and take it down. The autoscaler will happily do this unless max-instances bounds it. So the concurrency and autoscaling settings are not just about the service itself; they are the governor on the load the service is allowed to impose on the rest of the system, which makes tuning them an architectural decision rather than a local optimization.
The architecture: every piece explained
Concurrency is the per-instance request cap. It is the maximum number of requests Cloud Run will send to a single container instance at the same time. The default is a high value suited to I/O-bound web services; setting it to one reproduces classic per-request isolation. This number is the target the autoscaler steers toward: if concurrency is 80 and there are 800 concurrent requests, the platform wants roughly ten instances.
The autoscaler is the control loop. It continuously observes the number of requests each instance is handling and compares it to the concurrency target (in practice it aims to keep utilization comfortably below the hard cap so bursts have headroom). When incoming load pushes existing instances toward their concurrency limit, it starts new instances; when load falls, it drains and removes them. Uniquely among many serverless platforms, its floor is zero — with no traffic and no min-instances, the service scales to no instances at all and costs nothing.
Cold start is the cost of that scale-to-zero property. When a request arrives and no instance is available — either because the service was at zero or because every existing instance is at its concurrency limit — Cloud Run must start a new instance: pull or reuse the container image, boot the container, run the application's startup code, and only then dispatch the request. That startup latency is added to the first request the new instance serves. For a lightweight container it is small; for one with heavy initialization (loading a model, warming caches, establishing pools) it can be seconds.
min-instances is the warm-pool control that fights cold starts. Set it above zero and Cloud Run keeps that many instances running even when there is no traffic, so requests during a lull or at the leading edge of a burst hit a warm instance instead of paying a cold start. The trade-off is explicit: warm instances cost money while idle. max-instances is the opposite bound — the ceiling on how many instances the autoscaler may create. It exists to protect downstream dependencies from being overwhelmed and to cap the bill; when the ceiling is hit, additional requests queue or are rejected rather than spawning unbounded capacity.
CPU allocation underpins all of this. In the default request-time allocation, an instance is granted CPU only while it is actively handling requests; between requests its CPU is throttled, which is cheap but means background work and pre-warming stall when idle. In always-allocated (CPU always on) mode, the instance keeps its CPU between requests, which is necessary for background processing, keeps connection pools and caches alive, and smooths latency — at a higher cost. Concurrency, autoscaling behavior, and cold-start feel all shift depending on which allocation mode a service runs in.
End-to-end flow
Follow a burst of traffic into a service configured with concurrency 80, min-instances 1, and max-instances 20. During a quiet period one warm instance sits idle (kept alive by min-instances). A marketing email goes out and requests begin arriving. The first requests land on the warm instance, which starts serving them concurrently; no cold start is paid because the instance was already running.
Contrast that same burst on a service configured with concurrency one — the classic per-request model. Here the warm instance can serve only a single request at a time, so the moment a second request arrives the autoscaler must start a second instance, and every additional simultaneous request forces another cold start until the fleet catches up. The early part of the burst is therefore dominated by cold-start latency, and the instance count rises far faster than in the high-concurrency case for the same traffic. That is the right behavior for a CPU-bound handler that cannot share a core, but for an I/O-bound service it is pure waste — the same requests that one high-concurrency instance would have absorbed with room to spare instead trigger a swarm of cold starts and a much larger bill. Watching these two configurations diverge under identical load is the clearest illustration of why the concurrency knob must match the workload's CPU profile.
As requests accumulate, the single instance approaches its concurrency target of 80 simultaneous requests. The autoscaler sees utilization climbing and begins starting additional instances before the first is saturated, so there is headroom. Each new instance pays a cold start — image already cached, container boots, startup code runs, connection pools initialize — and then begins accepting its own share of the concurrent load. Within a few seconds the fleet has grown to match demand, packing roughly 80 requests onto each instance.
The burst intensifies and demand would call for more than 20 instances' worth of capacity. The autoscaler hits max-instances and stops growing. Now the existing 20 instances are at or near their concurrency ceiling, and further requests wait in Cloud Run's internal queue for a slot to open on an instance; if the wait exceeds the request timeout, those requests fail. This is the deliberate back-pressure that max-instances provides: the service refuses to scale past the point where it would overwhelm its database, rather than taking the whole system down.
The email campaign tapers off. Instances finish their in-flight requests and the autoscaler, seeing utilization fall, begins draining and shutting down instances — waiting for each to complete its outstanding requests before terminating it. Eventually traffic returns to a trickle and the fleet shrinks back toward one instance, the min-instances floor, which stays warm for the next burst. Across the whole episode the service paid for capacity only while it was needed, cold starts were confined to the growth phase, and the database was shielded by the max-instances ceiling — the concurrency and autoscaling settings together shaping cost, latency, and blast radius.