Tune Passive Outlier Detection Without Ejecting Too Many Hosts

One bad host, five percent of requests failing

The search service at Belgavi's Book Store runs on six hosts behind an Envoy proxy. One host loses its connection to the shared cache after a network change and starts failing requests that need the cache, about a third of its traffic. Active health checks keep passing because the health endpoint does not touch the cache. Customers see a little over five percent of searches fail, a share no single dashboard makes obvious.

Passive outlier detection is the mechanism for this case. The Envoy documentation calls it passive health checking: instead of probing hosts, the proxy watches the responses of real requests and ejects hosts that behave unlike the others. The settings below are Envoy's, read from its documentation and from the v1.40.2 source; other proxies have similar mechanisms with different names.

Consecutive errors

The simplest detector counts consecutive failures per host. consecutive_5xx, default 5, ejects a host after that many 5xx responses in a row. In the default mode, locally originated errors such as timeouts and connection resets count too. Detection runs inline, as responses arrive. consecutive_gateway_failure counts only 502, 503 and 504, but its enforcement percentage defaults to 0, so it detects without ejecting unless you turn it on.

Consecutive counting misses partial failure. The host above fails one request in three, interleaved with successes, so it never reaches five in a row and is never ejected. A host that has died outright is caught within five requests.

Success rate and failure percentage

Every interval, default 10 seconds, Envoy compares hosts. Success-rate detection considers hosts with at least success_rate_request_volume requests in the interval (default 100), and runs only when at least success_rate_minimum_hosts of them qualify (default 5). It computes the mean and standard deviation of their success rates and ejects hosts below mean minus a factor times the standard deviation. The factor is success_rate_stdev_factor divided by 1,000, default 1.9. The source comment works an example: rates of 50, 100, 100, 100 and 100 give a mean of 90, a standard deviation of 20 and a threshold of 52, which is 90 minus 1.9 times 20, so the 50 percent host is ejected.

Success rate is relative, which cuts both ways. With five hosts at 99.9, 99.9, 99.9, 99.9 and 80 percent the threshold is about 80.8 and the bad host goes. With two hosts at 80 percent the mean drops to 91.9, the deviation rises to 9.75, the threshold falls to 73.4 and neither bad host is ejected. When every host fails equally, because a shared dependency is down, nothing is an outlier and nothing is ejected, which is the right outcome. Failure-percentage detection compares each host with a fixed threshold instead (default 85 percent, volume 50, five hosts), but its enforcement also defaults to 0.

The cap that ejects nothing

Before ejecting, Envoy checks max_ejection_percent, default 10 percent. The v1.40.2 source computes the share as if the host were already ejected, 100 times (ejected hosts plus one) divided by the number of hosts, and ejects only if that is at most the limit.

So the six-host search cluster, with default settings, will detect the bad host and refuse to eject it; the documentation counts this as an overflow. Either raise max_ejection_percent to a value that admits at least one host, such as 20 for five to ten hosts, or set always_eject_one_host, which the API reference says ejects at least one host regardless of the percentage. The cap itself is a safety feature: it is what stops a detector from removing most of a cluster during a broad failure.

How long a host stays out

The ejection time is base_ejection_time (default 30 s) multiplied by the number of times the host has been ejected in a row, up to max_ejection_time (default 300 s, or the base if larger). A host that fails again right after returning stays out 60 s, then 90 s and so on. The multiplier decreases by one at each interval in which the host was healthy and not ejected, so the documentation's rule of thumb is that it takes about max over base times interval, 100 seconds with the defaults, to return to the base. A jitter of up to max_ejection_time_jitter spreads returns so ejected hosts do not all come back at once.

If active health checking is also configured, a successful active check returns the host to rotation and clears its counters, unless successful_active_health_check_uneject_host is false. In the opening example that would undo every ejection, because the active check never touches the cache. Turn it off or make the active check exercise the dependency.

A configuration for six hosts

Envoy cluster outlier_detection settings for the example; adjust volumes to your traffic.

outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_time: 300s
  max_ejection_percent: 34          # admits 2 of 6 hosts: (1+1)/6 = 33.3 percent
  success_rate_minimum_hosts: 5
  success_rate_request_volume: 100
  success_rate_stdev_factor: 1900
  successful_active_health_check_uneject_host: false

With these values the companion lab ejects the partially failing host at 40 s, the first interval check after the fault starts at 30 s, and again for 60 and then 90 seconds as it keeps failing after each return. Its traffic moves to the other five, and the failed-request rate during the fault falls from 5.5 percent with the default cap to 0.8 percent.

When ejection makes things worse

• Overload looks like failure. A host that returns 503 because it is overloaded is ejected, and its traffic goes to the others, which overload in turn. The cap limits how far this spreads; a cap of 100 percent removes that limit.
• Small clusters. Ejecting one of three hosts moves a third of its peers' load at once. Prefer success-rate detection with enough volume, and capacity for N minus one.
• Client-caused errors. A bad request that some hosts reject with 500 instead of 400 makes those hosts look unhealthy. Map responses correctly before trusting the detector.
• Shared clusters. The documentation warns that an ejection by one filter chain affects every chain that uses the cluster.
• Panic mode. If ejections and failed health checks leave fewer than the panic threshold of healthy hosts, 50 percent by default, Envoy balances across all hosts again, including ejected ones.

Verification

Watch the outlier detection statistics per cluster: ejections detected, enforced and overflowed. An overflow count that rises while a host is bad is the signature of the cap problem above. Test with fault injection on one host at a partial error rate, and confirm the ejection, the traffic shift and the return after the ejection time. The companion lab replays these settings with synthetic traffic and shows each decision.

Practice and related reading

Hands-on: Passive outlier ejection: consecutive errors, success rate and the cap, an interactive lab on synthetic data.

Further reading:

• Health Checks That Take a Fleet Down
• Service Mesh Traffic Management, in depth
• Metastable Failures: When Retries Keep a Service Overloaded

Sources and scope

Checked against these sources on 3 November 2026. Thresholds and caps are computed from the formulas in the Envoy v1.40.2 source; traffic figures come from the deterministic lab model, not from a measurement.

• Envoy documentation: Outlier detection (architecture overview)
• Envoy API reference: config.cluster.v3.OutlierDetection (fields and defaults)
• Envoy v1.40.2 source: outlier_detection_impl.cc (ejection cap, success-rate threshold, backoff)
• Envoy documentation: Panic threshold