Per-feature degradation ladders through a scripted provider outage

What this experiment demonstrates

Belgavi's Book Store serves 120 recommendation, 60 help-desk and 10 refund-eligibility requests a minute. A scripted 60-minute timeline sets the primary provider's error rate and p95 latency: an error spike at minutes 10-12, slow minutes at 15-17, errors at 18-19, a full outage at 20-39 and a ragged recovery at 40-46. A controller reads the previous minute, trips to degraded mode on the error or latency threshold and returns after a run of healthy minutes.

The secondary provider and the small model have per-minute capacity shared in proportion to demand, the secondary fails a share of calls during the outage, and the small model gets 15 percent of refund decisions wrong on a synthetic eval. All numbers are fixtures; nothing is called.

Run controlled comparisons

Start with the defaults: full ladders, refunds kept off the small model, a 20 percent error trigger, a 10 second latency trigger and a 3 minute hold. The controller trips at minute 11 and returns at minute 48. 4,212 requests are served by the primary, 6,618 by a fallback model, 476 from cache, rules or templates and 94 get an honest message (57 help-desk, 37 refunds queued for a person); no raw errors, no wrong refunds, no flapping.

Now change one thing at a time. Set all three ladders to primary only: 4,819 raw errors and 20 minutes with a feature fully down. Remove only the refund ladder: 252 raw errors and the same 20 minutes. Allow refunds on the small model: 25 wrong refund decisions; add the kill switch for recommendations and it falls to 3. The kill switch alone takes honest messages from 94 to 0. Set the hold to 1 minute: 6 extra mode switches from flapping. Raise the latency trigger to 30 seconds: 2 extra switches. Set the hold to 10: the primary serves 2,896 requests instead of 4,212. Turn off drills: honest messages rise to 503, and a cache hit rate of 80 brings them down to 186. A secondary quota of 0 gives 635 messages; a short help-desk ladder gives 1,175.

Inspect the trace and keep evidence

The table counts the requests each rung served for each feature over the hour, and the line below it shows the mode by minute. The trace lists every trip, return and flap and summarises each scripted phase, including how far demand exceeded the secondary quota and the small model's capacity. Start trace, Next step, Show all and the slider control how much of the trace is visible. Reset restores the defaults and Export JSON downloads the inputs, metrics and trace.

Limits and production translation

Real outages are not scripted, traffic shifts during them, caches go stale and quality on each rung varies by request. The controller reads one minute of signals with one minute of lag. What carries over: plan a ladder per feature, keep decisions with a cost off weaker models, size fallback capacity for every feature degrading at once, trip fast and return slowly, give each feature a kill switch, and drill every rung.