Deploying Agentic Systems

An agent run is a job, not a request

Deploying an agent with the same architecture you'd deploy a normal API service -- a stateless container behind a load balancer, autoscaled on request rate, request killed and retried on timeout -- breaks on contact with what an agent run actually looks like: it can take seconds or it can take twenty minutes, it accumulates state across many tool calls that shouldn't be thrown away on a transient failure, and if it misbehaves it can take an action with real consequences, not just return a wrong HTTP response. This article works through the deployment-architecture decisions that follow from taking those differences seriously.

Sandboxing execution

An agent that can execute code, run shell commands, or make arbitrary tool calls needs an execution boundary between what it can touch and what it can't -- the same principle as permission boundaries, applied at the infrastructure layer rather than the credential layer.

Container-per-run gives each agent run its own fresh, isolated container: strongest isolation (a compromised or buggy run can't affect another run, because there's no shared state to affect), but container startup latency (typically hundreds of milliseconds to a few seconds, depending on image size and cold-start handling) is pure overhead on every run, and idle capacity sits unused between runs unless the orchestration layer is tuned to scale containers down aggressively.

A shared sandbox pool (a warm pool of pre-started, reusable execution environments, reset between runs) amortizes the startup cost across many runs, at the price of needing to actually trust the reset step -- any state that leaks across a reset (a lingering process, a modified filesystem, a cached credential) is a cross-run contamination bug, and unlike container-per-run, that class of bug is silent until something specifically goes looking for it. Pools are the right choice when run volume is high enough that per-run container startup becomes a real cost or latency line item; container-per-run is the right default otherwise, because its failure mode (slightly higher latency) is far less dangerous than the pool's (silent state leakage).

Why agent runs break request-response autoscaling

A normal service autoscales on a proxy for load (requests per second, CPU) because request duration is roughly uniform and short -- ten thousand requests per second at 50ms each is a predictable, steady amount of concurrent work. An agent run's duration varies by two orders of magnitude depending on how many steps the task actually needs, so "requests per second" stops being a useful proxy for how much concurrent capacity is actually in use; ten agent runs started in the same second could mean ten seconds of total work or ten minutes, and the autoscaler needs to react to concurrent in-flight runs and their actual resource consumption, not arrival rate.

The practical fix is to treat agent execution as a job queue rather than a request-response endpoint: an API call enqueues a run and returns a handle immediately (or streams progress), a worker pool pulls from the queue and executes, and the worker pool autoscales on queue depth and worker utilization -- the same shape as a batch-processing system, not a web service, because that's what an agent run's duration profile actually resembles.

Blast-radius containment

Deployment architecture is the last line of defense when an agent's own judgment fails -- it hallucinates a destructive action, follows an injected instruction from untrusted content, or simply has a bug. The design question is: given that this specific run does something wrong, how much damage can it actually do, independent of whether the agent's own logic was supposed to prevent it.

Network egress restrictions on the execution sandbox (allowlist only the specific external endpoints a run's declared tools actually need, not open internet access) turn "the agent tries to exfiltrate data or call an unexpected external service" from a silent success into a blocked, loggable event. Filesystem restrictions (a fresh, ephemeral filesystem per run, with only explicitly mounted paths writable) turn "the agent modifies something it shouldn't" into "the agent modifies a throwaway copy." A hard wall-clock and resource ceiling per run (kill and flag anything that runs far longer or consumes far more than the task profile expects) catches the case where an agent gets stuck in a loop rather than letting it run indefinitely.

None of these replace the guardrail and permission-scoping logic that belongs in the agent's own design -- they're the layer that still holds when that logic has a gap, which is the actual justification for defense in depth here: the two layers fail independently, so a single mistake in one doesn't mean total exposure.

Checkpoint persistence across long-running work

A twenty-minute agent run that gets killed by a deploy, a node eviction, or a worker crash at minute nineteen should not have to restart from minute zero -- both because it's wasteful (re-doing eighteen minutes of tool calls and model calls that already succeeded) and because some of those earlier steps may have had real side effects (a message already sent, a record already written) that re-running would duplicate.

The infrastructure requirement this implies: durable, incremental checkpointing of run state (completed steps, accumulated facts, current position in the plan) to storage outside the worker process itself, so a new worker picking up the run after a crash can resume from the last checkpoint rather than from scratch. This is the deployment-layer half of what a planner's own state management needs on the application side -- see agent checkpointing for the application-level durable-execution patterns this infrastructure needs to support (resumability, replay, and treating side-effecting steps as needing idempotency keys so a resumed run doesn't repeat a real-world action).

Worker crash handling needs to distinguish "resume from checkpoint" from "retry from scratch" explicitly -- a worker that can't tell the difference will, on any crash, either wastefully restart safe-to-repeat work or dangerously repeat side-effecting work, and getting this distinction wrong in either direction is the most common production incident in agent deployment infrastructure.

Human-approval gates in the deployment path

For actions above a defined risk threshold, the deployment architecture itself -- not just the agent's own judgment -- should require an explicit approval step before the action executes, structurally: the tool call that would perform the action is intercepted by the execution layer, held pending approval, and only released to actually run once a human (or a separate, more constrained review process) approves it. This differs from the agent merely being instructed to ask permission, because an instruction can be skipped by a model that gets it wrong; a structural gate at the execution boundary can't be talked past by anything the model generates.

The design trade-off is latency and friction against safety, so the threshold matters: gating every tool call defeats the purpose of an autonomous agent, while gating nothing removes the safety net entirely. A blast-radius classification per tool (read-only, reversible write, irreversible/high-consequence) determines which calls need a gate, and only the last category typically warrants one in a mature deployment.

Sizing worker capacity in practice

A concrete example makes the queue-and-worker-pool shape easier to size than the abstract description does. Say a workload runs 500 agent triggers per hour, with a duration distribution of 30 seconds median and 8 minutes at the 95th percentile (a long tail from the small share of tasks that genuinely need many steps). Sizing workers off the median alone under-provisions badly for the tail; sizing off the 95th percentile alone over-provisions for the common case. The right approach is the same one queueing systems have always used: size steady-state worker count off a weighted average of the actual duration distribution and target concurrency, then add headroom (typically 20-40%) for the tail, and let the autoscaler handle burst above that headroom by adding workers on rising queue depth rather than by front-loading capacity for a worst case that's rare.

Queue depth itself becomes the primary autoscaling signal precisely because it directly reflects the thing that matters (how much work is waiting versus how fast it's being drained), where request rate would have reflected only how fast work is arriving -- a queue-depth-driven autoscaler naturally absorbs the duration variance that a request-rate-driven one can't.