RELIABILITY · LIVE EXPERIMENT
Circuit breaker: fail fast and recover deliberately
Break a dependency, watch the gate protect it, and decide when traffic can safely return.
AICASSINDRA · DECISION MISSION
Recovery Detective: Protect the Dependency, Then Recover
The dependency fails from 8 through 30 simulated seconds. Compare the same seeded traffic with the breaker enabled and bypassed. Your target at 60 seconds: at most 20 dependency failures and the first healthy response within 3 seconds after recovery.
Evidence: deterministic teaching simulation. Predict a result, compare the alternatives, then explain which constraint mattered. No model API or real infrastructure is contacted.
Run the mission to inspect a deterministic comparison.
Need a hint?
A short cooldown sends more probes during failure. A longer one can leave healthy requests rejected after the dependency recovers. Recovery also depends on where the probe schedule falls relative to second 30; it need not change monotonically.
Model assumptions and research reference
This mission replays the original seeded circuit-breaker engine: 12 requests/s, fixed 0.6-second responses, one half-open probe, consecutive failures to open, and one successful probe to close. There are no retries, jitter, network calls or production traffic. Fast rejections remain unsuccessful client requests. Dependency protection and successful client service are distinct outcomes. The comparison is a simulation, not an observed service benchmark.
Amazon Builders’ Library: avoid amplifying failures. The mission and its numerical targets are original teaching designs.
Explain the surprise: which result looked better until you counted the missing cost? Change one input and find a counterexample to your first conclusion.
0 requests generated · 0 recovery probes
Your experiment
Compare fast rejections with dependency failures. Inspect the cooldown, the single half-open probe, and in-flight requests that finish after the gate opens.
- Run the scheduled outage and inspect the state history.
- Pause, change one control, and step forward.
- Compare a second run using the same seeded traffic.
Successes + dependency failures + fast rejections + in-flight requests must equal requests generated.
Decision trace
Model assumptions: one shared breaker, consecutive-failure trip threshold, fixed response time, one half-open probe, and one successful probe to close. No retries or production network calls. Scheduled outage runs from 8s to 30s; changing the error rate switches to manual control. Rejected requests are not successful responses. Pattern reference.
Distinguish protection from successful work
A circuit breaker reduces repeated attempts against a failing dependency. It does not turn rejected work into successful responses. This experiment follows individual generated requests into four mutually exclusive outcome buckets: succeeded, failed at the dependency, rejected by the breaker, or still in flight. Their sum equals requests generated. The animated gate, counters and state history come from that request model rather than a manual state toggle. The dependency is entirely fictional and no network calls are made.
Set traffic, failure and recovery controls
Requests per second controls generated demand. Dependency errors is the probability sampled for a new request using a repeatable pseudorandom sequence. Moving this slider selects manual failure control instead of the scheduled outage. Consecutive failures to open counts failed responses while closed; a successful response resets the streak. Response time sets the fixed duration captured by new attempts, not requests already in flight. Cooldown before probing sets the duration of future open periods; it does not change a deadline already armed. Enable circuit breaker switches between protected and bypassed traffic.
Reproduce an outage and inspect the first recovery attempt
Outage + recovery starts a new trace that is healthy until eight seconds, fails until thirty seconds, and becomes healthy afterward. With defaults, inspect fifteen seconds: the breaker has rejected requests and is open or probing. Step to forty-five seconds and observe healthy closed traffic again. Closed admits requests. Open rejects new requests until the cooldown elapses. Half-open admits one probe while other requests are rejected. One successful probe closes this selected policy; one failed probe opens it for another cooldown. This differs from implementations that require multiple successful probes or a rolling failure percentage.
Show why pending calls do not disappear
Increase Response time and run the outage. Several calls can already be in flight before enough failed responses have arrived to trip the breaker. Those calls still complete and contribute to success or failure counters after opening. Their stale state-epoch results do not alter the new breaker state. The model samples a call’s success or failure at admission and delivers it after its captured latency. Repairing the service does not rewrite outcomes of calls already admitted. There are no cancellation, retry or timeout layers hidden behind the counters.
Compare bypass and manual repair
Choose Healthy to reset the trace without a scheduled outage. Disable the breaker, select Break service, and step ten seconds: requests continue reaching the dependency and fail instead of being rejected by a protective gate. Enable the breaker to compare its effect. Repair service sets future failure probability to zero, but an already open breaker still waits for its armed cooldown before probing. Healthy and Outage + recovery reset the trace while retaining the selected rate, threshold, response time and enabled setting; they are scenario presets, not universal resets of every control.
Pause, rewind and branch the recorded history
Play runs for up to ninety seconds. Pause freezes the current state. Step +1s advances twenty fixed model ticks. Restart begins a fresh trace with the currently selected configuration and schedule mode. Speed offers half, normal, double and four-times viewing pace. Replay recorded history can visit only times that have already been simulated. After rewinding, resuming, stepping or changing a model control creates a new branch and discards the later recorded future. The replay cursor restores the selected configuration together with request state, so the controls describe the historical snapshot being viewed.
Interpret the model without production guarantees
The SVG uses moving request and response markers, while text counters and the decision trace provide equivalent state evidence. The visible decision list retains recent transitions; the recorded state timeline and counters remain inspectable through replay. One shared breaker, fixed latency and seeded independent failures simplify a real dependency. There is no concurrency limit, exponential backoff, distributed breaker coordination, retry budget, rolling error window or measured availability. Changing model controls while playing affects subsequent ticks. Fast rejection may reduce dependency load but remains an unsuccessful client request in this exercise.