Core concept: Linux cgroups as resource isolation
YARN containers are Linux processes, and without OS-level isolation, one greedy task can starve the entire cluster by hogging CPU or memory that other containers on the same node need. Linux control groups (cgroups) are the kernel's answer: a process can be placed into a cgroup and the kernel enforces hard limits on the memory it consumes, the CPU bandwidth it gets, and the I/O it performs. YARN uses cgroups to ensure that a container's resource consumption is bounded no matter how badly its workload behaves.
The isolation strategy is straightforward: when the NodeManager launches a container, it creates a new cgroup hierarchy for that container and attaches the container's JVM process to it. The kernel then enforces the resource limits defined for that cgroup. If a container tries to allocate more memory than its limit, the kernel kills the process. If a container's CPU usage exceeds its allocated bandwidth, the kernel throttles it. This is hard isolation: no amount of aggressive garbage collection or spinning threads inside the container can break out and steal resources from its neighbors.
Cgroups are organized hierarchically on the filesystem, typically mounted at /sys/fs/cgroup. Each resource (memory, CPU, I/O, network) gets its own subdirectory, and YARN creates container-specific cgroups within each. For example, a container might appear in /sys/fs/cgroup/memory/hadoop-yarn/container_1234_001/ with a memory.limit_in_bytes file that enforces the memory ceiling. When the NodeManager initializes on a node, it must verify that cgroups are mounted and configured correctly — if they are not, container isolation fails silently and the node becomes unsafe for multi-tenant workloads.
How it works: the LinuxContainerExecutor and cgroup hierarchy
YARN's LinuxContainerExecutor is the component that manages cgroups on the NodeManager. When a container is allocated to a node, the executor constructs a cgroup path, creates the cgroup in the kernel, sets resource limits, and then executes the container's process within that cgroup. The setup requires several pieces to work in concert.
First, cgroups must be mounted on the operating system. On a fresh Linux box, cgroups are usually in the rootfs but not yet used by containers; the operator must enable and mount them explicitly or use a tool like cgrulesengd. YARN expects the hierarchy to be present at boot time — it does not create mount points, only the cgroups within the existing hierarchy. A typical configuration mounts memory and cpu cgroups under /sys/fs/cgroup.
Second, the LinuxContainerExecutor needs elevated privileges. To create cgroups and move processes between them, it must run as root or have setuid privileges. The typical deployment pattern is to run the executor as a setuid binary: the Hadoop admin compiles the executor with special permissions so that when the NodeManager (which runs as the Hadoop user) invokes it, the executor escalates to root for the privileged operations and then drops back to the application user. This is complex to set up correctly because getting the privilege boundary wrong opens security holes.
Third, cgroup limits are set via pseudo-files. For memory, the executor writes the container's allocated memory (e.g., "4294967296" for 4 GB) to memory.limit_in_bytes. For CPU, it writes the number of CPU cores and a period to cpu.cfs_quota_us and cpu.cfs_period_us. For example, to allocate 1 full CPU core, it might set quota to 100000 microseconds and period to 100000 microseconds. The kernel then throttles the process group if they exceed this quota in a period.
When a container terminates, the executor removes it from the cgroup, deletes the cgroup hierarchy, and releases the resources. If the container was killed by the kernel (e.g., due to OOM), the executor can still clean up the cgroup.
The configuration happens in yarn-site.xml. The key properties are:
<property> <name>yarn.nodemanager.container-executor.class</name> <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value> </property> <property> <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name> <value>hadoop-yarn</value> </property> <property> <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name> <value>/sys/fs/cgroup</value> </property> <property> <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name> <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value> </property>
The cgroups.hierarchy sets the root name for YARN's cgroup subtree (e.g., all YARN containers live under /sys/fs/cgroup/*/hadoop-yarn/). The cgroups.mount-path tells YARN where the kernel mounted cgroups. The resources-handler.class selects the cgroup resource handler; CgroupsLCEResourcesHandler is the standard choice for modern kernels.
Cgroups v1 and v2 are different. Most production YARN clusters still use cgroups v1, which has separate hierarchies for each resource controller (memory, cpu, cpuacct, devices, etc.). Cgroups v2 unifies them into a single hierarchy but requires kernel 4.5+ and is less tested in Hadoop deployments. The LinuxContainerExecutor primarily targets v1.
Configuration reference and setup checklist
Enabling cgroups requires changes at three levels: the kernel, the filesystem, and the Hadoop configuration. A minimal setup checklist:
Kernel and filesystem: Verify that your kernel supports cgroups (most modern kernels do). Check that the cgroup filesystem is mounted: mount | grep cgroup should show lines for memory, cpu, and other controllers. If not, you can mount them manually with mount -t cgroup -o memory cgroup /sys/fs/cgroup/memory or configure them in /etc/fstab for persistence.
Executor binary: Compile or obtain the setuid LinuxContainerExecutor binary. In a Hadoop distribution, it is typically at $HADOOP_HOME/bin/container-executor. Verify its permissions: ls -la $HADOOP_HOME/bin/container-executor should show -rwsr-xr-x (with the setuid bit). If the bit is missing, run chmod u+s $HADOOP_HOME/bin/container-executor as root.
Hadoop configuration: Add the properties shown earlier to yarn-site.xml. Additionally, set container memory and CPU allocations in the ResourceManager's scheduler configuration (CapacityScheduler or FairScheduler). For example, in capacity-scheduler.xml, set yarn.scheduler.capacity.memory-maximum to the maximum container size.
Permissions and ownership: The executor binary must be owned by root with setuid. The cgroup directories created by YARN should be readable by the Hadoop user and writable by root. Some operators use a hadoop group and make the executor group-owned by hadoop to simplify permissions.
After setup, restart the NodeManager and verify that containers are placed into cgroups. Check the NodeManager logs for messages like cgroup or CgroupsLCEResourcesHandler. If cgroups are working, you should see lines indicating successful cgroup creation for each container.
Monitoring, debugging, and common failure modes
When a container is running, inspect its cgroup with Linux tools. For a container ID like container_1626567890_001, find its cgroup path: find /sys/fs/cgroup -name "*container_1626567890_001*". Then examine the resource limits and usage:
cat /sys/fs/cgroup/memory/hadoop-yarn/container_1626567890_001/memory.limit_in_bytes cat /sys/fs/cgroup/memory/hadoop-yarn/container_1626567890_001/memory.usage_in_bytes cat /sys/fs/cgroup/cpu/hadoop-yarn/container_1626567890_001/cpu.cfs_quota_us
If the usage is close to the limit, the container is under memory pressure. If the limit is much smaller than expected, check the YARN configuration. A common mistake is setting per-container memory limits too conservatively in the scheduler, then wondering why containers are killed prematurely.
Common failure modes:
Cgroups not mounted. Error: Cannot find cgroup mount. Solution: Mount cgroups explicitly or verify the mount in /etc/fstab.
LinuxContainerExecutor not setuid. Error: Permission denied when creating cgroups. Solution: Run chmod u+s $HADOOP_HOME/bin/container-executor as root and verify with ls -la.
Container killed by OOM. Error: Container exits with exit code 137. Solution: Increase the container memory allocation or add swap (not recommended) to the node. Check whether the application is genuinely memory-hungry or if there is a leak.
Stale cgroups consuming memory. Symptoms: Kernel memory usage slowly increases over weeks. Solution: Manually clean up stale cgroups with rmdir /sys/fs/cgroup/*/hadoop-yarn/container_* 2>/dev/null (carefully!) or enable automatic cleanup via cgroup-cleanup scripts.
CPU throttling with no indication. Symptoms: Tasks take longer than expected but do not fail. Check the cgroup's cpu.stat for nr_throttled and throttled_time_ns. If these are non-zero, the container is being throttled.
Best practices for production
Always enable cgroups on multi-tenant clusters. The risk of a single noisy container is too high. Test cgroup setup in a staging environment before deploying to production.
Allocate memory with headroom. If an application uses 2 GB at peak, allocate 2.5 or 3 GB per container to account for bursty allocation patterns and GC overhead. The cost is slightly lower utilization; the benefit is fewer OOM kills.
Use separate cgroup hierarchies for different workloads if possible. If you run both batch and interactive workloads, consider giving batch jobs their own cgroup subtree so that interactive queries are not affected by batch container isolation failures.
Monitor cgroup metrics actively. Set up custom metrics collection to export memory and CPU usage per container to your observability platform (Prometheus, Grafana, etc.). This helps diagnose container behavior and sizing issues quickly.
Plan for cgroup cleanup. Schedule a weekly job to remove stale cgroups: find /sys/fs/cgroup -type d -name "container_*" -mtime +7 -delete (with careful testing first). Or configure systemd's DefaultMemoryAccounting and MemoryMax to handle cleanup.
Document your setup. Cgroup configuration is not obvious. Document which kernel version is required, which filesystem mounts are needed, which Hadoop configuration properties are set, and which binaries need which permissions. Future operators (including you, in 6 months) will thank you.
Test failure recovery. Periodically simulate node crashes or container executor failures and verify that cgroups are cleaned up and the cluster recovers. Do not assume that cleanup is automatic — it usually requires manual intervention or a restart.
Trade-offs and operational gotchas
The big win: noisy-neighbor prevention. In a multi-tenant cluster, cgroups are essential. Without them, a single container with a memory leak can consume node memory until other containers are evicted by the kernel's OOM killer, potentially killing critical workloads. With cgroups, the leaking container hits its memory limit and dies alone, while others continue unaffected. Similarly, a CPU-bound container that spins in a tight loop without cgroups can starve I/O-bound containers on the same node; with cgroups, it gets its allocated CPU share and no more. This is the reason to enable cgroups on any shared cluster.
But the setup is notoriously complex. Getting cgroups right requires kernel support, correct filesystem mounting, the LinuxContainerExecutor compiled and installed with setuid permissions, and the Hadoop configuration tuned correctly. Many operators discover this the hard way: they enable cgroups, the cluster works fine for a few days, and then a node crash or kernel upgrade reveals that the cgroup mount is not persistent or the executor binary lost its setuid bit. The failure mode is insidious because containers still run — isolation just silently fails.
Permission requirements are particularly tricky. The LinuxContainerExecutor must run as setuid root, which means the binary is a security boundary. If it has a vulnerability, attackers can escalate privileges. The Hadoop team mitigates this by keeping the executor minimal and carefully reviewing its code, but the risk remains non-zero. Some operators avoid it by running the NodeManager as root directly, but this is even riskier because the entire Java process gains root access.
Memory limits create sharp cliffs. When a container hits its memory limit, the kernel's out-of-memory killer immediately terminates it. There is no gradual degradation or spillover to disk — the process is killed and the task fails. This is intentional (to prevent one container from bringing down the node) but it means that workloads with bursty memory usage (e.g., certain Spark operations) can be fragile if the memory allocation is too tight. Many operators add headroom — allocating 1.5x the peak memory they observe — to avoid hair-trigger OOM kills.
CPU throttling is more forgiving but can surprise users. When a container exceeds its CPU bandwidth limit, the kernel does not kill it; instead, it sleeps the process until the next period. This means the container continues to run but at a reduced effective speed. A task that expected to use 4 cores and allocated only 2 will complete in roughly twice the time, not fail outright. This is less obvious to users and can lead to cascading delays (job A misses its deadline, causing job B to miss its deadline, etc.).
Cgroup filesystem is mutable and manual. The resource limits live in pseudo-files on the cgroup filesystem. If an operator (or a rogue script) manually edits these files, they can change limits on running containers. This is rarely a problem in practice because most operators do not edit cgroups by hand, but it means the cgroup filesystem is not truly immutable. YARN has no auditing of cgroup changes, so if someone manually increases a container's memory limit, YARN will not know and may later assign that freed capacity to another container, oversaturating the node.
Cgroup hierarchies can leak. If a node crashes or the LinuxContainerExecutor fails to clean up a container, the cgroup may persist on the filesystem. After reboot, stale cgroups linger and can consume small amounts of kernel memory. Over months of operation, a node with dozens of stale cgroups can accumulate hundreds of megabytes of kernel memory pressure. The mitigation is to periodically clean up stale cgroups manually or use a cgroup cleanup tool, but many operators do not realize this is necessary.
CPU accounting is approximate. The CPU bandwidth allocation via cpu.cfs_quota_us and cpu.cfs_period_us is enforced by the kernel scheduler, but it is not perfectly fair across all workloads. The kernel tries to throttle fairly, but if a container has many short-lived threads, the scheduler's granularity might allow brief periods where it uses more than its allocation. Additionally, the CPU time reported in the cgroup's cpuacct subsystem can lag the actual consumption by a few milliseconds because it is sampled, not perfectly precise. This rarely matters in practice but it means you cannot use cgroup CPU metrics as a perfect audit trail.
Docker / container images add another layer. If YARN containers run Docker containers, there are now two levels of cgroup isolation: the YARN cgroup and the Docker container's cgroup. If they are not configured to agree (e.g., YARN allocates 4 GB to a container but Docker's cgroup limit is 2 GB), the more restrictive limit wins. Mismatches can be hard to debug because the Docker container appears to run out of memory before the YARN allocation limit is reached, leading to confusion about who is responsible for the limit.
Monitoring and visibility are poor. YARN does not automatically expose cgroup metrics (like actual memory usage or CPU time per container) in its web UI or metrics system. To see whether a container is being throttled or approaching its limits, you must manually inspect /sys/fs/cgroup or use Linux tools like systemd-cgtop. Some operators wire up custom monitoring to parse cgroup files and export metrics to their observability stack, but this is not a built-in feature.
The decision to enable or disable cgroups affects stability. Some operators disable cgroups because the setup is complex, gambling that their cluster is trusted and single-tenant enough not to need isolation. This works until a developer accidentally runs an infinite loop in production, at which point the entire node becomes unusable. Other operators enable cgroups but misconfigure them, leading to OOM kills that they blame on Hadoop rather than their own infrastructure. Getting it right requires understanding Linux kernel behavior, YARN architecture, and operational discipline.