Node memory pressure: kubelet eviction versus kernel OOM

What this experiment demonstrates

A node runs three pods: a log shipper, an API server whose memory grows, and a pricing service. Two different systems can reclaim memory, and they choose different victims. The kubelet watches node-level signals and evicts a whole pod when a threshold is crossed. The kernel out-of-memory killer fires when memory is exhausted faster than the kubelet reacts, or when a container exceeds its own limit, and it kills a process by an internal score. The experiment makes the API server grow and shows which mechanism fires first and which pod it takes.

You set the node memory and the hard eviction threshold, the API server request, limit and growth rate, the log shipper request (zero makes it BestEffort), and whether the growth is a gradual climb or a sudden spike, along with the kubelet reaction lag. The QoS class of each pod follows from its request and limit, and that class steers both reclaimers.

Run controlled comparisons

Start with the defaults: an 8192 MiB node, a 100 MiB hard threshold, the API server requesting 2048 and limited to 4096 MiB growing at 60 MiB per second, a BestEffort log shipper, and gradual growth. The API server climbs toward its 4096 MiB limit and is OOM-killed inside its own cgroup before the node runs short, so the first event is a container OOMKilled restart, not node pressure: adding nodes would not help.

Now change one thing. Remove the API server limit (set it to zero) so it can grow past 4096: in this fixture the node crosses the eviction threshold first and the kubelet evicts, and because the log shipper is BestEffort it is the victim while the Guaranteed pricing pod survives. Turn on the spike, or raise the growth or the reaction lag until the growth outruns the kubelet: the kernel acts first and kills by oom_score at the process level instead of a clean eviction. With the limit still removed, give the log shipper a request so it becomes Burstable and under its request: the victim shifts to the API server, because it is now the pod most over its request. The fixture usually preserves the pricing pod; Guaranteed QoS is not immunity from eviction or node OOM.

Inspect the trace and keep evidence

The trace names each pod with its QoS class and then shows which mechanism fired, at what time, and why it chose its victim. The metrics report the first reclaim, the time to the event, OOMKilled restarts, pods evicted, whether a critical Guaranteed pod was killed, and whether the pricing pod survived. Start trace, Next step, Show all and the slider control the visible trace; the metrics and the pod table always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, the metrics, the pod table and the trace.

Limits and production translation

This memory-only model uses a simplified QoS/usage victim heuristic, not exact kubelet priority ordering or kernel oom_score. CPU settings are fixed assumptions: qualifying equality for Guaranteed cases and absent CPU resources for BestEffort. It grows one pod linearly and reclaims instantly; real accounting and scheduling differ. Real nodes have many pods, bursty allocation, and eviction policies whose hard thresholds can use zero grace. What carries over: set memory requests on everything that matters, for Guaranteed QoS set equal CPU requests/limits and equal memory requests/limits on every app and init container; this memory-only fixture assumes those CPU settings for its Guaranteed cases, leave BestEffort only for work that is safe to lose, reserve memory headroom to give the kubelet time to react; this cannot guarantee it beats a sudden allocation spike, and investigate whether an OOMKilled restart came from a container limit or node pressure rather than inferring the cause from the status alone.