Metastable Failures: When Retries Keep a Service Overloaded After the Trigger Is Gone

An outage that outlives its cause

The order-status API at Belgavi's Book Store handles about 80 requests a second on capacity for roughly 100. One evening a database failover slows it to 40 percent capacity for twenty seconds. The failover finishes, the database is healthy, the API servers are healthy, and the API stays down. Every request times out. Restarting the servers helps for a few seconds, then it is down again. It recovers only after the team blocks traffic at the load balancer for a minute.

Bronson, Aghayev, Charapko and Zhu named this a metastable failure in their HotOS 2021 paper. Once load rises above an implicit threshold, a system enters a vulnerable state: still healthy, not overloaded, but one trigger away from a bad state with unusably low goodput, which a sustaining effect, often work amplification, keeps in place after the trigger is removed. The follow-up study by Huang and colleagues at OSDI 2022 found such failures in incident reports from many organisations, including at least 4 of 15 major AWS outages in the decade they reviewed, and described different kinds of triggers and of amplification.

The sustaining loop

In the order-status case the loop has three parts.

1. The queue outlives the trigger. During the slowdown, requests arrive faster than they are served and a backlog builds. When capacity returns, the backlog is still there, and every request waits behind it.
2. Work completes too late to matter. Once the wait exceeds the client timeout, the server spends its capacity on requests whose clients have already given up. Goodput, the work that arrives in time, falls to zero even though the servers are busy.
3. Retries multiply the load. Each timed-out request is retried, so offered load becomes new requests plus retries. With 3 retries, 80 requests a second can become up to 320 attempts a second against capacity for 100. The backlog never drains.

The loop sustains itself without the trigger. That is the defining property, and it is why restarting servers does not help: the load, not the servers, is in the bad state.

A simulation: when it recovers and when it does not

The companion lab models exactly this: one-second steps, 80 new requests a second, capacity 100, a 5-second client timeout, up to 3 retries with exponential backoff starting at 1 second, and a capacity drop of 60 percent starting at 60 seconds. It is a teaching model with deterministic arrivals, not a model of any particular system.

Three things stand out. The boundary is sharp: in this model an 11-second slowdown is absorbed and a 12-second one is permanent, which is why these incidents look disproportionate to their causes. Longer backoff alone does not save the system; it spreads the retries but does not reduce their number enough. And the most effective change is cheap: when the server discards requests whose deadline has passed instead of processing them, wasted work disappears and the backlog drains at once.

Breaking the loop

• Propagate deadlines and drop expired work. Send the client deadline with the request, as gRPC does with deadlines, and check it when a request is dequeued and before expensive steps. Work for a caller who has gone is pure waste.
• Bound retries with a budget. Allow retries up to a fraction of recent successful or new requests, in the spirit of gRPC retry throttling and the AWS SDK retry quota, so retries stop when most requests are failing.
• Retry at one layer. Three layers retrying three times each turns one request into as many as 64 attempts.
• Bound the queue. A queue limit keeps waits short enough that completions remain useful; rejected requests fail fast and cheaply.
• Shed load early and by priority. Admission control that rejects a fraction of new work when queueing delay rises keeps goodput up for the rest.
• Have a reset lever. Know how to remove load quickly, such as a load balancer rule or a feature flag, because once the loop is running, removing load is the only fix.

Dropping expired work in code

Illustrative python, checked for syntax; adapt names and storage to your system.

import time

def worker_loop(queue, handle, now=time.monotonic):
    """Each item carries the absolute deadline its client sent; skip work nobody will receive."""
    while True:
        item = queue.get()
        if now() >= item.deadline:
            item.discard(reason='deadline passed in queue')   # cheap: no database call
            continue
        remaining = item.deadline - now()
        handle(item, timeout=remaining)                     # pass the budget to every downstream call

The check costs almost nothing and turns wasted work into a fast discard, which is what the lab models with its drop option. Clocks differ between machines, so propagate a remaining budget in each hop, as gRPC deadlines do, and convert it to a local monotonic deadline on arrival.

Detecting the vulnerable state

Metastability is a property of the system under load, not of a component, so it does not show up in unit tests or ordinary health checks. Signals that the system has entered or is near the bad state:

• Goodput falls while server utilisation stays high: the servers are busy doing work nobody receives.
• Attempts per unique request rise above 1 for longer than the trigger lasted.
• Queueing delay exceeds the smallest client timeout in the call path.
• Recovery requires removing load, and restarts do not help.

Test for it deliberately: inject a capacity drop in a load test at production-like utilisation, remove it, and check that goodput returns. If it does not, find the amplification mechanism before production does.

Failure modes of the fixes

• Deadlines that are not checked. Passing a deadline without checking it at dequeue time changes nothing.
• Retry budgets per client instance. A thousand clients each allowed 10 percent retries still add up to the same amplification; budgets work best where they see aggregate failure rates.
• Caches as hidden amplifiers. A cache that empties during the trigger turns hits into backend reads; the look-aside cache example in the HotOS paper shows how that alone can sustain the failure, because slow database responses stop the cache from refilling.
• Autoscaling as the cure. New capacity takes time to start and may be overwhelmed by the backlog on arrival; it raises the threshold but does not remove the loop.

Practice and related reading

Hands-on: Metastable overload: a short slowdown, retries and a backlog, an interactive lab on synthetic data.

Further reading:

• Retry, exponential backoff and jitter, with retry budgets
• Backpressure architecture: bounded queues and credit-based flow control
• Load shedding and admission control for agent-to-agent systems

Sources and scope

Checked against these sources on 1 November 2026. Recovery figures come from the companion lab model with deterministic arrivals; they illustrate the mechanism and are not measurements of a real service. Outage counts are quoted from the OSDI 2022 paper.

• Bronson, Aghayev, Charapko, Zhu: Metastable Failures in Distributed Systems (HotOS 2021)
• Huang et al.: Metastable Failures in the Wild (OSDI 2022)
• gRPC documentation: Deadlines
• gRPC documentation: Retry (retry throttling)
• AWS SDKs and Tools Reference Guide: Retry behavior (retry quota)