Why architecture matters here
Architecture matters here because the failure a breaker prevents is not the obvious one. Everyone plans for a dependency returning an error; the error is easy — you catch it, you fall back, you move on. The dangerous failure is the dependency that accepts your connection and then takes thirty seconds to answer, or never answers at all. Each such call holds a worker thread hostage for the full timeout, and under concurrency the pool drains faster than calls complete. Once the pool is empty, even requests bound for perfectly healthy dependencies queue behind the stuck ones. The whole agent goes dark because of one sick backend.
A breaker attacks this at the source: it stops the stuck calls from being made in the first place. When it observes that recent calls to a dependency are failing or timing out at a high rate, it opens and rejects new calls to that dependency immediately, in microseconds, without consuming a thread on a doomed wait. The threads that would have been hostages stay free to serve other work. This is the difference between a localized brownout of one tool and a total outage of the agent.
The pattern also does the callee a favor. A dependency that is struggling — a database under lock contention, a rate-limited API, an overloaded model endpoint — recovers faster when the flood of retrying callers stops hammering it. By shedding load from a failing backend, the breaker gives it the breathing room to catch up, rather than holding it underwater with a steady stream of requests it cannot serve. The breaker is a cooperative back-pressure signal, not just a selfish shield.
For an agent specifically, the breaker turns a class of failures into a decision. When the model endpoint is open, the agent does not simply crash; it takes the fallback branch — return a cached answer, escalate to a human, degrade to a simpler non-LLM path, or tell the user honestly that a capability is temporarily unavailable. That branch is a design surface: because the breaker makes the failure explicit and fast, the agent author gets to choose a graceful response instead of inheriting a hung request. Resilience becomes something you author rather than something you hope for.
Finally, breakers compose with the rest of the stability toolkit in a way that is more than additive. A timeout bounds how long any single call may wait; a bulkhead bounds how many calls to one dependency may be in flight at once; a breaker bounds whether calls are attempted at all given recent history. Timeout limits the damage of one slow call, bulkhead limits how many slow calls can coexist, and the breaker limits how long the agent keeps trying before it gives up on the dependency entirely. Each covers a different failure geometry, and an agent that wants to stay up under partial outages needs all three.
The architecture: every piece explained
The heart of the diagram is the three-state machine. In the CLOSED state the breaker is transparent: calls pass straight through to the dependency, and the breaker's only job is to record the outcome of each one — success, failure, or a slow call that exceeded its latency threshold — into a sliding window. As long as the recent failure ratio stays below the configured limit, the circuit remains closed and traffic flows normally. Closed is the healthy steady state; the breaker earns its keep only when things go wrong.
When the failure ratio in the window crosses the failure-rate threshold, the breaker transitions to OPEN. In the open state, every call to the dependency is rejected instantly — the breaker throws a call-not-permitted signal without touching the backend. This is the load-shedding phase: the dependency gets a rest, and the agent's threads are not consumed on hopeless waits. The open state is timed; a reset timeout defines how long the breaker stays open before it is willing to test the waters again.
When the reset timeout elapses, the breaker moves to HALF-OPEN. Here it admits a small, bounded number of trial calls — the probe budget — and watches them closely. If the trials succeed, the dependency appears healthy and the breaker transitions back to CLOSED, resuming full traffic. If the trials fail, the dependency is still sick, and the breaker snaps back to OPEN for another timeout period. Half-open is the recovery gate: it prevents the breaker from dumping the full traffic load onto a backend that has not actually recovered.
The sliding window is what makes the trip decision meaningful. It can be count-based (the last N calls) or time-based (all calls in the last T seconds), and it tracks not just failures but slow calls — those that returned successfully but took longer than a latency threshold — because a dependency that answers slowly is often on its way to answering not at all. Tripping on slow-call rate, not just error rate, catches the degradation before it becomes a full outage.
Two companion controls round out the picture. The bulkhead and timeout box bounds each dependency independently: a cap on concurrent in-flight calls so one dependency's slowness cannot consume every thread, and a per-call timeout so no single call waits forever — the timeout is also what feeds the slow-call counter. The fallback path is the answer the agent gives when the circuit is open: a cached response, a degraded computation, a graceful message. Together these turn a tripped breaker from a dead end into a controlled detour, and the ops strip below tracks whether the whole arrangement is tuned: how often the breaker trips, how long it stays open, how often the fallback is exercised, and whether half-open probes are succeeding.
End-to-end flow
Follow one tool call through the breaker as the dependency degrades. Under normal conditions the circuit is CLOSED. A user turn asks the agent to look something up; the agent invokes the tool, the call passes through the breaker to the backend, returns in forty milliseconds, and the breaker records a success in its window. Hundreds of such calls flow, the failure ratio sits near zero, and the breaker is invisible. This is the case that must stay cheap: in the closed state the breaker adds only a window update per call.
Now the backend starts to struggle — a dependency of its own is slow, and its latencies climb from forty milliseconds toward the per-call timeout. The breaker's window begins filling with slow-call and timeout outcomes. Each timed-out call is failed fast at the timeout boundary rather than hanging indefinitely, so threads are returned to the pool, but the failure ratio in the window is climbing. The agent is still serving, but the breaker is watching the ratio approach the threshold.
The ratio crosses the threshold and the breaker trips to OPEN. From this instant, calls to the tool are rejected immediately without touching the backend. The agent's call site catches the call-not-permitted signal and takes the fallback branch: perhaps it returns a slightly stale cached result, perhaps it tells the user this particular lookup is temporarily unavailable and continues with the rest of the task. Crucially, no threads are being spent waiting on the sick backend anymore, so the agent's capacity to serve every other session and every other tool is fully preserved. The blast radius is contained to the one dependency.
The backend, no longer being hammered, begins to recover. Meanwhile the breaker's reset timeout counts down. When it elapses, the breaker moves to HALF-OPEN and admits a handful of trial calls — say the probe budget is three. The next three tool invocations are allowed through while all others still get the fast rejection. If those three succeed within the latency threshold, the breaker concludes the dependency is healthy and transitions back to CLOSED, and full traffic resumes. If even one or two of the trials fail, the breaker returns to OPEN and waits out another timeout before probing again — protecting the still-fragile backend from a premature flood.
Step back and see what the sequence bought. During the degradation, the agent never exhausted its thread pool, because slow calls were bounded by the timeout and then eliminated by the open circuit. During the outage, users got a graceful degraded experience instead of hung requests, because the fallback path was authored and the failure was fast. During recovery, the backend was not re-flooded, because half-open admitted only a probe budget. And throughout, every transition was observable — trip, open duration, half-open outcome — so operators could see exactly what happened and tune the thresholds. All of that falls out of one small state machine wrapped around one dependency.