Phi-accrual failure detectors, in depth: per-peer state, Cassandra and Akka internals, pause guards and threshold calibration

Per-peer state

Every node runs one detector per peer it monitors. In a fully gossiping cluster that is every other member. In Akka Cluster, each node is watched by a bounded set of observers (monitored-by-nr-of-members defaults to 9), so heartbeat traffic does not grow quadratically. The per-peer state is small:

• Last arrival time, taken from a monotonic clock. Wall-clock time is wrong here because an NTP step can move it backwards and produce a negative interval or a huge one.
• A bounded window of inter-arrival intervals, plus running sums of the intervals and their squares, so the mean and variance cost O(1) per update instead of a pass over the window.
• The last heartbeat version seen, where heartbeats travel indirectly. Cassandra peers bump a heartbeat counter in their gossip state, and an observer records an arrival when it learns a newer value, whichever peer relayed it.

So in Cassandra, intervals measure gossip propagation, not a direct round trip. B can look healthy because other peers relay its fresh counter while your node cannot reach it. Phi there answers "is B alive somewhere?", not "can I talk to B?", so track direct reachability separately through request success rates.

Two production models

Both implementations compute phi from the same idea but make different modelling bets. Cassandra's FailureDetector keeps up to 1,000 samples per endpoint and assumes exponentially distributed intervals. Under that model the probability of a gap longer than t is exp(-t/mean), so phi is simply t / (mean x ln 10). The code computes t / mean and multiplies by a constant PHI_FACTOR = 1/ln(10), about 0.434, before comparing against phi_convict_threshold (default 8). The source comments say the factor exists to keep that historical default meaningful. Intervals above a cap, which defaults to the detector's initial value of twice the gossip interval, are not recorded, so a partition does not teach the window that a 30-second gap is normal.

Akka's PhiAccrualFailureDetector assumes normally distributed intervals and evaluates the normal tail with a logistic approximation. Two settings shape it. min-std-deviation (default 100 ms) sets a floor on the standard deviation, because a quiet LAN can produce an almost zero variance and then a single 50 ms hiccup would push phi past any threshold. acceptable-heartbeat-pause (default 3 s) is added to the mean before phi is computed, which buys tolerance for GC pauses and brief network drops at the cost of slower detection. The defaults are a 1 s heartbeat interval, threshold 8.0 and 1,000 samples. The window is seeded with two synthetic intervals around the configured heartbeat-interval (mean plus and minus a quarter of it), so the first real heartbeat does not start from an empty history.

The exponential model is memoryless and has no variance term, so it is robust and slow. A one-second gossip cadence convicts after roughly 18 seconds at threshold 8. The normal model is sharp: when jitter is small it convicts within a few seconds of the expected arrival, which is why Akka adds the pause allowance.

A complete detector

The implementation below keeps both models behind one interface, uses running sums, applies an interval cap and a standard deviation floor, and returns phi as a float so callers own the decision.

import math, time
from collections import deque

class PhiDetector:
    def __init__(self, model="normal", max_samples=1000, first_estimate_ms=1000,
                 min_std_ms=100, pause_ms=3000, max_interval_ms=None):
        self.model, self.min_std, self.pause = model, min_std_ms, pause_ms
        self.max_interval = max_interval_ms           # None = record everything
        self.win = deque(maxlen=max_samples)
        self.s = self.s2 = 0.0
        self.last = None
        sd = first_estimate_ms / 4                    # seed, as Akka does
        for v in (first_estimate_ms - sd, first_estimate_ms + sd):
            self._push(v)

    def _push(self, v):
        if len(self.win) == self.win.maxlen:          # evict oldest from the sums
            old = self.win[0]; self.s -= old; self.s2 -= old * old
        self.win.append(v); self.s += v; self.s2 += v * v

    def heartbeat(self, now_ms=None):
        now = now_ms if now_ms is not None else time.monotonic() * 1000
        if self.last is not None:
            gap = now - self.last
            if self.max_interval is None or gap <= self.max_interval:
                self._push(gap)
        self.last = now

    def phi(self, now_ms=None):
        if self.last is None:
            return 0.0
        now = now_ms if now_ms is not None else time.monotonic() * 1000
        t = now - self.last
        n = len(self.win); mean = self.s / n
        if self.model == "exponential":               # Cassandra style
            return t / (mean * math.log(10))
        sd = max(math.sqrt(max(self.s2 / n - mean * mean, 0.0)), self.min_std)
        y = (t - (mean + self.pause)) / sd
        e = math.exp(-y * (1.5976 + 0.070566 * y * y))
        if t > mean + self.pause:
            return -math.log10(e / (1.0 + e))
        return -math.log10(1.0 - 1.0 / (1.0 + e))

The running sums drift slightly in floating point over millions of updates, so recompute them from the window every few thousand evictions.

Worked example: when each detector fires

Take an Akka-style detector with a 1 s heartbeat whose observed jitter is tiny, so the 100 ms floor applies. The effective mean is 1,000 + 3,000 = 4,000 ms. Phi reaches 8 when the tail probability is 1e-8, which the logistic approximation hits at about y = 5.2 standard deviations. That is 4,000 + 520, about 4.5 seconds after the last heartbeat. If the link has 200 ms of real jitter, the same threshold fires near 5.0 seconds, so the detector loosens itself on a noisier path without a config change.

Now a Cassandra-style detector with mean gossip arrival of 1 s. Phi is t / 2.303, so phi 8 needs t of about 18.4 seconds, and phi 5 about 11.5 seconds. The two systems use the same threshold number and convict roughly four times apart, because the number only has meaning relative to its model.

Do not read phi 8 as "a one in 100 million chance of a false positive". That holds only if intervals really follow the model, and they do not. Real inter-arrival distributions have heavy tails from GC, kernel scheduling, retransmits and noisy neighbours, so the true false-positive rate at phi 8 is many orders of magnitude higher than 1e-8. Treat phi as a calibrated ranking of surprise, then measure the actual false-positive rate yourself.

The observer pause problem

The most damaging false positive is self-inflicted. If the observer is frozen for 10 seconds by a stop-the-world collection, a hypervisor migration or a suspended container, then on waking it sees every peer as 10 seconds late. Every detector it runs crosses the threshold at once, and it may gossip DOWN states for the whole cluster. Cassandra guards against this. It tracks the gap between its own detector ticks, and when that gap exceeds a maximum local pause it skips conviction for that round. The same idea in code:

def check_all(detectors, threshold, state):
    now = time.monotonic() * 1000
    stalled = state.last_tick is not None and now - state.last_tick > state.max_local_pause_ms
    state.last_tick = now
    if stalled:
        return []                     # we were frozen; silence says nothing about peers
    return [peer for peer, d in detectors.items() if d.phi(now) > threshold]

A second guard is a correlation check. If more than a configured fraction of peers cross the threshold in the same tick, the likeliest explanation is local: a stall, a NIC problem or a partition on this side. Hold conviction and raise an alert instead of declaring half the cluster dead. Partition handling is covered in split-brain prevention.

Calibrating thresholds from traces

Thresholds should come from your own traffic. Calibration has three steps. First, record arrival timestamps per peer for a week, including deploys, compactions and backups, because those are when false positives happen. Second, replay the traces through the detector at candidate thresholds. Count false suspicions (phi crossed while the peer was known to be healthy) and measure detection time on injected crashes, which you add by cutting traces at random points. Third, pick the knee of that curve for each consumer.

def replay(trace_ms, threshold, crash_at=None, tick_ms=100, **kw):
    d, false_pos, detect = PhiDetector(**kw), 0, None
    arrivals = [t for t in trace_ms if crash_at is None or t < crash_at]
    i, now, suspected = 0, arrivals[0], False
    while now < trace_ms[-1] + 60_000:
        while i < len(arrivals) and arrivals[i] <= now:
            d.heartbeat(arrivals[i]); i += 1; suspected = False
        if not suspected and d.phi(now) > threshold:
            suspected = True
            if crash_at is not None and now >= crash_at:
                detect = now - crash_at; break
            false_pos += 1
        now += tick_ms
    return false_pos, detect

Summarise false suspicions per peer-day and detection time p50/p99 across all peers. The router may accept one false suspicion per peer-week for fast detection; failover wants none and accepts tens of seconds. Two thresholds, one stream.

Failure modes

Most phi incidents in practice fall into a small set of patterns.

The gray-failure row is the subtle one. Heartbeats come from a lightweight thread that can stay punctual while the data path is drowning, so a node can be useless and still score phi 0. Make heartbeats share fate with real work where you can, and read gray failure for detection strategies beyond heartbeats.

Operating phi in production

Export phi itself, not only the boolean, for every monitored peer. A heatmap of phi by peer over time shows the difference between one node dying (one row goes bright), a network event (a block of rows rises together) and an observer stall (every row from one observer spikes at once). Alongside it, graph the window mean and standard deviation per peer, conviction and recovery events, and the time from conviction to the action each consumer took.

Change thresholds one consumer at a time, and keep the heartbeat interval fixed while you do: a new interval moves the window's mean, so phi is not comparable until about 1,000 intervals have refilled the window. A detector only suspects. Any action that cannot tolerate a live old owner, such as promoting a new primary, also needs fencing tokens. Gossip dissemination of the resulting state is covered in gossip membership.

Trade-offs

Phi accrual earns its complexity when the network is heterogeneous and several consumers need different risk levels. It is harder to explain than a timeout, a slowly degrading link teaches it to accept degradation, and it only sees heartbeats. A fixed timeout is still reasonable for a two-node system on one switch, and SWIM-style probing with indirect pings scales better when you need direct reachability across thousands of members. Many systems combine them: phi for suspicion, probes to confirm, and fencing to make the final action safe.

What to do next

1. List every consumer of failure information in your system and write down the cost of a false suspicion and of a slow detection for each.
2. Confirm that every arrival time uses a monotonic clock and that a local-pause guard exists.
3. Record per-peer arrival traces for a week that includes deploys and maintenance.
4. Replay them through replay() at several thresholds and choose one per consumer from the false-suspicion versus detection-time curve.
5. Export phi per peer as a metric and build the heatmap before you need it in an incident.
6. Add a correlated-suspicion hold so that one observer cannot convict most of the cluster in a single tick.
7. Put fencing in front of every action that assumes the old owner is gone.