Node Memory Pressure on Kubernetes: Kubelet Eviction Thresholds, the Kernel OOM Killer, QoS Scores and Which Pod Dies First

Two reclaimers, two victims

When a Kubernetes node runs low on memory, two separate systems can act, and confusing them leads to the wrong fix. The first is the kubelet, which watches node-level signals and evicts whole pods when a threshold is crossed; this is node-pressure eviction, a graceful, policy-driven process. The second is the Linux kernel out-of-memory killer, which fires when memory is exhausted faster than the kubelet can react, and kills individual processes by an internal score with no knowledge of pods. The kubelet evicts; the kernel kills. They use different thresholds and choose victims by different rules, so a pod can be OOM-killed while the kubelet reports the node as healthy, and a pod can be evicted while plenty of RAM appears free.

How the kubelet chooses

The kubelet compares signals such as memory.available against configured eviction thresholds. A soft threshold waits a grace period and lets the pod terminate normally; a hard threshold, which defaults to memory.available<100Mi, acts immediately. When it must evict for memory pressure, the kubelet ranks pods first by whether their memory use exceeds their requests, then by Pod Priority, then by how far usage is over the request. That ranking is why requests matter: a pod using less than it requested is the last to be chosen, and a pod with no memory request looks like it is always over, so it is among the first.

How the kernel chooses

If memory is exhausted before the kubelet evicts, the kernel acts. Under cgroup v2 each container runs in a memory cgroup, and a container that exceeds its memory limit is OOM-killed within its own cgroup regardless of node pressure, which is the common cause of a single container restarting with reason OOMKilled on an otherwise healthy node. When the whole node is out of memory, the kernel picks a victim by its oom_score, which Kubernetes biases through oom_score_adj: Guaranteed pods get the most protective value, BestEffort the least, so the kernel and the kubelet tend to agree on who should die, but the kernel acts at the process level and can take a critical process inside a pod without evicting the pod cleanly.

Illustrative yaml, checked for syntax; adapt names and storage to your system.

# Make the important pod Guaranteed: requests == limits for memory.
apiVersion: v1
kind: Pod
metadata: {name: pricing}
spec:
  containers:
  - name: pricing
    image: pricing:1.4
    resources:
      requests: {memory: "512Mi", cpu: "500m"}
      limits:   {memory: "512Mi", cpu: "500m"}
# memory.high can be set via the cgroup to throttle before the hard limit,
# giving the kubelet time to act before the kernel OOM-kills the container.

A worked example

A node has 8 GiB allocatable. Three pods run: a BestEffort log shipper with no requests, a Burstable API server requesting 2 GiB with a 4 GiB limit, and a Guaranteed pricing service with memory requests and limits both 512 MiB and equal positive CPU requests and limits on every container. A memory leak in the API server pushes it toward its 4 GiB limit. Two outcomes are possible, and which one happens depends on speed.

The pricing service survives every scenario because it is Guaranteed and under its request. The lesson is not that one mechanism is better but that requests and limits steer both: they set the kubelet ranking and the kernel score at the same time.

Reading the signals apart

The two mechanisms also show up differently in what you can observe, and confusing the signals leads to the wrong remedy. A kubelet eviction appears as an event on the node and on the pod, with a reason naming memory pressure, and the pod is gone and may be rescheduled elsewhere. A container OOM kill appears as a restart of that one container, with the last state reporting an OOM reason and the restart count climbing, while the pod itself stays scheduled on the same node. A node-level kernel kill is the harshest to read, because it can take a process without a clean pod event at all, so you see a workload misbehave with little explanation on the Kubernetes side and the evidence sits in the kernel log instead. Treating these as one number, a generic restart count, hides which of three different problems you have.

Configuring a node so the right pod dies first

1. Set memory requests on everything that matters. A pod with no request is first to be evicted and least protected from the kernel.
2. Make genuinely critical pods Guaranteed by setting memory limits equal to requests, so both reclaimers protect them last.
3. Leave BestEffort only for work that is safe to lose, such as a log tail or a cache warmer.
4. Tune eviction headroom to allow reaction time; hard eviction may use zero grace and sudden spikes can still trigger kernel OOM.
5. Watch for OOMKilled restarts separately from evictions; they mean a container hit its own limit, which is a sizing problem, not node pressure.

Failure modes

• Treating an OOMKilled container restart as node pressure and adding nodes, when the fix is a larger memory limit or a memory leak to find.
• Running important workloads BestEffort because no one set requests, so they are the first victims of both reclaimers.
• Setting a memory limit far above the request, which makes the pod Burstable and killable while looking generously sized.
• A hard eviction threshold at or near zero, so the kernel OOM killer always beats the kubelet to the reclaim.
• Assuming eviction is graceful for the whole node; a fast spike skips it entirely and the kernel acts process by process.

Verification

In a test cluster, run a pod that allocates memory on command and watch which mechanism fires: a slow climb should trigger a kubelet eviction event on the lowest-QoS pod, while a fast allocation past the container limit should produce an OOMKilled restart of that container only. Confirm a Guaranteed pod survives both. Read kubectl describe node for eviction events and the container last-state for OOMKilled, and keep both as separate dashboard signals. The two counts answer different questions: evictions mean the node is under-provisioned for its pods, OOMKilled restarts mean a container is under-sized for its own workload.

Practice and related reading

Hands-on: Node memory pressure: kubelet eviction versus kernel OOM, an interactive lab on synthetic data.

Further reading:

• Kubernetes operator pattern
• Health check architecture
• Java in containers
• JVM heap tuning

Sources and scope

Checked against these sources on 23 October 2026. Examples and figures are illustrative, not measurements.

• Kubernetes documentation: Node-pressure eviction
• Kubernetes documentation: Pod Quality of Service classes
• Linux kernel: Control Group v2 (memory controller, memory.high, OOM)
• Kubernetes documentation: Controllers (control loops reconcile current to desired state)