Per-Feature Degradation Ladders: Planning Every Step Before an Outage

Three features, one outage

Belgavi's Book Store runs three features on the same model provider. Recommendations suggest the next book. The help desk drafts answers for cases such as CASE-5120, where Tanishka asks why her order of three paperbacks arrived as two. Refund eligibility decides whether Jenya's return of a damaged atlas worth ₹1,450 qualifies. When the provider starts failing, these three should not fail in the same way. A slightly worse recommendation is fine. A worse help-desk draft is acceptable if a person reviews it. A wrong refund decision costs money and trust, and it is the one most teams discover they never planned for.

The routing article on this site covers how to make a second route substitutable. This one is about the plan per feature: an ordered list of rungs written, reviewed and drilled before anything breaks.

The ladder

A degradation ladder lists, in order, what a feature does when the rung above it cannot serve. A typical full ladder has six rungs.

1. Primary model on the primary provider.
2. Secondary provider or model that passed the same evaluation set.
3. Smaller or self-hosted model, cheaper and weaker, with its own evaluated quality bar.
4. Cached answer for a request seen before, such as a popular question or yesterday's recommendations.
5. Rules or a template: a popular-books list, a canned answer for common intents, deterministic eligibility rules.
6. An honest unavailable message with another path: "Answers are delayed; open a help-desk case and Dinky will reply by email" or "Your refund request is queued for a person".

The AWS Well-Architected reliability pillar (REL05-BP01) frames this as turning hard dependencies into soft ones: a component should keep its core function when a dependency fails, possibly with stale or alternate data, and it calls this "as much a business discussion as a technical one". The Google SRE chapter on handling overload describes the same idea as serving degraded responses that are less accurate or contain less data. Neither treats the last rung as a failure; an honest message is a designed outcome, a raw error is not.

Features that must never degrade to a weaker model

Decide per feature which rungs are allowed, and write the reason down. For refund eligibility at the store, the ladder skips the small model entirely: primary, then a secondary that passed the same refund evaluation set, then deterministic rules that decide clear cases (within 30 days, photo attached), then a queue for a person. A refund that waits an hour is recoverable; a wrong approval or a wrong denial is not.

The companion lab makes the cost visible. When refunds may fall back to the small model, recommendations and the help desk crowd them out of the secondary's quota and the small model decides the overflow: 25 wrong refund decisions in one scripted hour, using a 15 percent error rate that is a fixture, not a measurement. Keep refunds off the small model and that number is zero; 37 refunds are queued for a person instead.

Trigger signals and hysteresis

Mode switching needs signals and a rule for coming back. Useful signals are the error rate of the primary, its latency against your own deadline, and quota exhaustion. The Claude errors reference read on 2 November 2026 shows why the type matters: a 529 overloaded_error is transient, while a 429 caused by reaching a monthly spend cap "has no retry-after header and keeps failing until access resumes", so retrying it is pointless and the ladder should move straight down.

Trip fast, return slowly. Return only after several consecutive healthy intervals, or a ragged recovery flips the mode back and forth, and every flip sends a burst of traffic to a provider that has just recovered.

Runnable with Python 3.11 or newer, standard library only; the comments show its output. Each letter is the mode chosen after observing that minute.

LADDERS = {
    'recommendations': ['primary', 'secondary', 'small_model', 'cache', 'popular_list', 'unavailable'],
    'help_desk': ['primary', 'secondary', 'small_model', 'cache', 'template', 'unavailable'],
    # Never a weaker model: the secondary passed the same refund eval set; then rules, then a person.
    'refund_eligibility': ['primary', 'secondary', 'eligibility_rules', 'queue_for_person'],
}


class ModeController:
    """Trip on the previous minute's signals; return only after `hold` healthy minutes."""

    def __init__(self, max_error_pct=20, max_p95_s=10, hold=3):
        self.max_error_pct, self.max_p95_s, self.hold = max_error_pct, max_p95_s, hold
        self.degraded, self.healthy_run, self.switches = False, 0, 0

    def observe(self, error_pct, p95_s):
        bad = error_pct >= self.max_error_pct or p95_s >= self.max_p95_s
        if not self.degraded and bad:
            self.degraded, self.healthy_run, self.switches = True, 0, self.switches + 1
        elif self.degraded:
            self.healthy_run = 0 if bad else self.healthy_run + 1
            if self.healthy_run >= self.hold:
                self.degraded, self.switches = False, self.switches + 1
        return 'degraded' if self.degraded else 'primary'


def first_rung(feature, mode):
    ladder = LADDERS[feature]
    return ladder[1] if mode == 'degraded' and len(ladder) > 1 else ladder[0]


recovery = [100, 100, 25, 5, 45, 5, 30, 5, 3, 1, 1]  # error % for minutes 38-48
for hold in (1, 3):
    ctl = ModeController(hold=hold)
    modes = [ctl.observe(err, 5)[0] for err in recovery]
    print('hold=%d  %s  switches=%d' % (hold, ''.join(modes), ctl.switches))
print(first_rung('refund_eligibility', 'degraded'))
# hold=1  dddpdpdpppp  switches=6
# hold=3  dddddddddpp  switches=2
# secondary

With a one-minute hold, the controller follows every wobble and switches six times in these eleven minutes, four more than needed; with a three-minute hold it switches twice: once down, once back.

Kill switches and drills

A kill switch is a flag that takes a feature off the model rungs on purpose. In the lab, turning off model rungs for recommendations during the incident frees the secondary's quota for the help desk and refunds: honest unavailable messages fall from 94 to 0, and the wrong refund decisions with the small model allowed drop from 25 to 3. Criticality-aware admission, as in the SRE overload chapter, can achieve the same automatically.

Rungs that never run do not work. The Amazon Builders' Library article on avoiding fallback warns that fallback logic is hard to test, is not exercised regularly and can carry latent bugs, and recommends running fallback paths continuously. The lab has a toggle for this: an undrilled small-model rung whose prompt references a removed field raises honest unavailable messages from 94 to 503. Drill each ladder on a schedule by forcing a feature to each rung in turn and checking its output against the same tests.

Lab results

The scripted hour has an error spike, slow minutes, a 20-minute outage and a ragged recovery. With defaults the controller trips at minute 11 and returns at minute 48; 4,212 requests are served by the primary, 6,618 by a fallback model and 476 from cache, rules or templates. Without the latency trigger, the slow minutes let it return early and it trips again three minutes later.

Failure modes

• A ladder for the service, not the feature. One global fallback sends refunds wherever recommendations go.
• Fallback capacity nobody sized. A secondary quota that fits one feature does not fit three moving at once.
• Silent degradation. Record the rung in every response and show it where it matters to the user.
• No exit criteria. Without hysteresis, recovery becomes a new incident.

Checklist

1. For each feature, list its rungs, the evaluation each rung passed and the rungs it may never use.
2. Size secondary and small-model capacity for all features degrading together.
3. Define trip signals by error type, latency and quota, and a hold period for returning.
4. Give every feature a kill switch and an honest final message with another path.
5. Drill each rung on a schedule and alert on the share of traffic served per rung.

Practice and related reading

Hands-on: Per-feature degradation ladders through a scripted provider outage, an interactive lab on synthetic data.

Further reading:

• Multi-Provider Model Routing and Graceful Degradation
• Circuit Breakers and Backpressure for Model APIs
• Capability Fallback Chains for Agents
• Designing an AI Incident Response Runbook
• Metastable Failures and Retry Amplification

Sources and scope

Checked against these sources on 2 November 2026. The controller output comes from running the snippet. The outage timeline, traffic, capacities, cache hit rate and the small model refund error rate in the lab are synthetic fixtures.

• AWS Well-Architected Reliability Pillar: REL05-BP01 Implement graceful degradation
• Google SRE book, chapter 21: Handling Overload (criticality, degraded responses)
• Google SRE book, chapter 22: Addressing Cascading Failures (deadlines, retries, graceful degradation)
• The Amazon Builders' Library: Avoiding fallback in distributed systems
• Claude API documentation: Errors (error types, errors after a 200, long requests)