Passive outlier ejection: consecutive errors, success rate and the cap

What this experiment demonstrates

The search service at Belgavi's Book Store runs behind a proxy with passive outlier detection. Traffic is spread evenly over the hosts in rotation for 300 seconds. From 30 s to 240 s, the faulty hosts fail a set share of their requests, interleaved with successes, so the longest run of consecutive failures follows from the rate. Consecutive-error ejection happens as responses arrive; success-rate ejection, returns to rotation and the decay of the ejection multiplier happen at each interval, in the order the Envoy source uses. The ejection cap uses Envoy's rule: a host is ejected only if the ejected hosts, counting it, are at most the configured percentage of the cluster.

It is a deterministic model of those rules with synthetic traffic, not a proxy. No network is used.

Run controlled comparisons

Start with the defaults: six hosts, one failing 33 percent of its requests, Envoy's default settings. The detector finds the outlier 21 times and the 10 percent cap refuses every ejection; 6,929 requests fail, 5.5 percent during the fault.

Now change one thing at a time. Raise max ejection percent to 34: three ejections, the first at 40 s, and 0.8 percent of requests fail during the fault. Turning on always eject one host instead gives the same result. With the cap at 34, turn success-rate detection off: nothing is ejected, because a host failing one request in three never reaches five consecutive errors. Then set the error rate to 100 percent: consecutive errors eject the host at 30 s. Go back to the defaults and use 10 hosts with 1,000 requests per second: the default cap now allows one ejection. With the cap at 34, make two hosts faulty: neither is ejected, because the mean and spread move with them. Make the fault hit every host: nothing is an outlier, and 33 percent fail. Finally set four faulty hosts at 100 percent, success rate off and the cap at 100: four hosts are ejected, fewer than half remain, and panic mode sends traffic to every host for 266 seconds.

Inspect the trace and keep evidence

The table lists ejections, refusals by the cap and returns to rotation with their times. Start trace, Next step, Show all and the slider control how much of the decision trace is visible. Reset restores the defaults and Export JSON downloads the inputs, metrics and trace.

Limits and production translation

Real traffic is uneven, failures cluster, the proxy adds jitter to ejection times and active health checks can return a host early; the model has none of these and does not model overload of the remaining hosts. What carries over: consecutive-error detection misses partial failure, success-rate detection is relative and needs volume and enough hosts, the default cap ejects nothing in a cluster of fewer than ten hosts, and ejecting too much triggers panic mode.