Core Concept
Hadoop node decommissioning is a graceful, controlled process for removing a DataNode from an active cluster without losing data or disrupting ongoing workloads. Unlike abrupt shutdowns, decommissioning ensures all blocks stored on the target node are replicated to other nodes before removal, maintaining the cluster's replication factor and data durability guarantees.
The decommissioning process is driven by the NameNode, which orchestrates block re-replication across healthy nodes. This is essential for hardware maintenance, cluster right-sizing, node retirement, and cloud-based environments where nodes are ephemeral. YARN supports an analogous decommissioning mechanism for NodeManagers via yarn.resourcemanager.nodes.exclude-path.
Why Decommissioning Matters
Without a controlled decommissioning process, shutting down a DataNode would result in immediate data loss if blocks on that node fell below the minimum replication factor (typically 3). The NameNode would mark those blocks as under-replicated, but the cluster's ability to re-replicate them would be compromised or impossible, especially in bandwidth-constrained environments or when other nodes are unavailable. Decommissioning solves this by: • Providing time for the NameNode to actively re-replicate all blocks from the target node to other nodes in the cluster • Maintaining data locality and rack diversity during re-replication • Allowing administrators to monitor progress and adjust cluster topology dynamically • Preventing cascading failures in already-stressed clusters • Supporting zero-downtime hardware maintenance and upgrades • Enabling planned capacity adjustments without operator intervention during the transition
Architecture and Concepts
The decommissioning process involves four key players: the NameNode, DataNodes, the Exclusion File, and the HDFS client layer.
The NameNode maintains a list of nodes marked for decommissioning via the exclusion file. It tracks the decommissioning state of each node (NORMAL, DECOMMISSION_INPROGRESS, DECOMMISSIONED) and actively schedules re-replication tasks for all blocks stored on nodes in the DECOMMISSION_INPROGRESS state.
DataNodes being decommissioned stop accepting new block writes but continue serving existing blocks until they are fully replicated elsewhere. They send heartbeats to the NameNode with updated block reports.
The exclusion file is a plain-text configuration file (typically specified via dfs.hosts.exclude) containing a list of hostnames or IP addresses, one per line. Nodes listed here are automatically transitioned to DECOMMISSIONING state.
Block re-replication follows the rack-awareness policy: the NameNode attempts to place replicas on nodes in different racks and, within a rack, on different nodes when possible. During decommissioning, this policy is relaxed slightly to prioritize completing re-replication over maintaining perfect rack diversity.
Configuration and Setup
To enable decommissioning in HDFS, configure the NameNode with the exclusion file path in hdfs-site.xml:
<property>
<name>dfs.hosts.exclude</name>
<value>/etc/hadoop/conf/excludes</value>
</property>
The exclusion file should be readable by the NameNode process and contain one hostname per line. For example:
datanode1.example.com
datanode2.example.com
10.0.0.15
After updating the exclusion file, refresh the NameNode's list of nodes without restarting:
hdfs dfsadmin -refreshNodes
This command triggers the NameNode to parse the updated exclusion file and begin transitioning listed nodes to DECOMMISSIONING state. You can verify the decommissioning status using:
hdfs dfsadmin -report
The output will show each DataNode's state: Live, Dead, or Decommissioning (with block counts and replication progress).
The Decommissioning Process
Decommissioning follows a well-defined progression that typically takes hours to days depending on cluster size, data volume, and network bandwidth.
Phase 1: Entry to Decommissioning
When a node is added to the exclusion file and hdfs dfsadmin -refreshNodes is executed, the NameNode marks it as DECOMMISSION_INPROGRESS. The node remains alive and responsive but will not accept new block writes. Existing replicas are served normally.
Phase 2: Block Re-replication
The NameNode scans all blocks on the decommissioning node and schedules re-replication tasks for each block that would fall below the minimum replication factor once the node is removed. It prioritizes blocks that are already under-replicated or at risk.
Re-replication respects the cluster's rack-awareness topology. If possible, the NameNode places new replicas on different racks to maintain fault tolerance. However, if insufficient nodes exist in other racks, in-rack replicas are used. The NameNode also throttles re-replication to avoid overwhelming the network with too many concurrent transfers.
Phase 3: Monitoring Progress
Administrators monitor decommissioning progress via hdfs dfsadmin -report, which shows the number of blocks remaining on each decommissioning node. Progress is typically slow on the last few blocks due to replication bottlenecks or missing target nodes.
Phase 4: Completion and Removal
Once all blocks have been successfully re-replicated, the NameNode marks the node as DECOMMISSIONED. At this point, the node can be safely shut down, removed from the cluster, or repurposed without any risk of data loss. The node can then be removed from the cluster configuration entirely.
Monitoring and Troubleshooting
Monitoring decommissioning progress is critical to understanding cluster health and identifying bottlenecks.
Key Metrics to Track
Use hdfs dfsadmin -report to view the decommissioning node's block counts. The key fields are:
• Under replicated blocks: blocks below the replication factor
• Blocks with corrupt replicas: damaged blocks that cannot be safely re-replicated
• Missing blocks: blocks with no surviving replicas (data loss)
Check the NameNode web UI (port 50070) for visual decommissioning status and per-node replication progress.
Common Issues and Solutions
If a decommissioning node gets stuck with remaining blocks, investigate:
1. Network connectivity: ensure the node and target nodes are reachable
2. Available disk space: target nodes must have space for new replicas
3. Rack constraints: if the cluster is heavily imbalanced across racks, re-replication may stall
4. Replication throttling: the NameNode may rate-limit re-replication to avoid overwhelming the network
If a decommissioning node hosts corrupt blocks or missing blocks, those blocks may never be re-replicated. In such cases, you can either:
• Accept the data loss and proceed with decommissioning
• Abort decommissioning, repair the corrupt data, and retry
• Manually remove the node if data loss is acceptable
To abort decommissioning, simply remove the node from the exclusion file and run hdfs dfsadmin -refreshNodes again.
Trade-offs + Gotchas
Decommissioning is not instantaneous and introduces several trade-offs and operational challenges. Time to Decommission Large nodes with terabytes of data can take days or weeks to fully decommission. Re-replication bandwidth is limited by network throughput, and the NameNode throttles re-replication to avoid overwhelming the cluster. Plan decommissioning well in advance and monitor progress. Network Overhead During decommissioning, the cluster experiences elevated network traffic as blocks are copied to new replicas. This can impact user-facing MapReduce and Spark jobs. Consider scheduling decommissioning during maintenance windows or periods of low cluster utilization. Rack Topology Constraints If your cluster has imbalanced rack topology (e.g., one rack with many nodes, another with few), decommissioning may stall when attempting to maintain rack diversity. The NameNode may reject re-replication placements that violate rack policies, leading to under-replicated blocks. Missing or Corrupt Blocks If a decommissioning node contains the only replica of a block (perhaps due to previous hardware failures), decommissioning cannot proceed without data loss. Similarly, corrupt blocks cannot be safely re-replicated. These situations require manual intervention. Hardware Refresh and Right-sizing Decommissioning is useful for hardware refresh (replacing aging nodes) and cluster right-sizing (reducing capacity). However, removing many nodes simultaneously can stress the cluster's re-replication capacity and network bandwidth. Decommission nodes serially or in small batches. Namespace Quotas If a target node for re-replication has a namespace quota or storage quota, re-replication may fail if the quota is already exhausted. Monitor quotas and increase them before decommissioning large clusters.
Best Practices
Following these practices ensures smooth, safe node decommissioning:
Plan Ahead
Before decommissioning, calculate the expected re-replication time based on node capacity, cluster size, and network bandwidth. Inform users about potential performance impacts and schedule decommissioning during maintenance windows.
Decommission Serially
Decommission one or two nodes at a time, never many in parallel. This allows the cluster to fully re-replicate blocks from one node before moving to the next, reducing network contention and avoiding cascading failures if new nodes fail during re-replication.
Verify Cluster Health
Before starting decommissioning, run hdfs dfsadmin -report and hdfs fsck / to identify missing, corrupt, or under-replicated blocks. Fix these issues first to avoid complications during decommissioning.
Monitor Re-replication Progress
Use the NameNode web UI and hdfs dfsadmin -report to track decommissioning progress hourly. Investigate stalls or errors immediately. Adjust replication throttling parameters (e.g., dfs.datanode.replication.streams.hard-limit) if needed.
Gracefully Handle Under-replication
If decommissioning stalls with remaining blocks, investigate and resolve the root cause. Only force-remove a node if data loss is explicitly acceptable.
Document Operations
Maintain a log of decommissioned nodes, dates, and reasons. This helps with capacity planning and troubleshooting.
YARN NodeManager Decommissioning
YARN's ResourceManager supports a similar decommissioning mechanism for NodeManagers via yarn.resourcemanager.nodes.exclude-path. This is configured identically to HDFS but controls container scheduling instead of block replication.
When a NodeManager is marked for decommissioning in YARN, the ResourceManager stops assigning new containers to it but allows existing containers to complete gracefully. Once all containers finish (or after a timeout), the NodeManager can be safely removed.
YARN decommissioning is typically faster than HDFS because containers are temporary, but it can still take time if long-running jobs are active. Coordinate HDFS and YARN decommissioning to avoid confusion: decommission the YARN NodeManager first, wait for all containers to finish, then decommission the HDFS DataNode.
Some clusters run both services on the same physical nodes, so coordinating the exclusion lists is important to ensure a smooth, parallel decommissioning process.
Summary
Hadoop node decommissioning is a critical operational procedure for removing DataNodes from an active cluster without data loss. It ensures all blocks are re-replicated to other nodes before removal, maintaining replication factor and data durability.
The process is driven by updating the exclusion file and refreshing the NameNode, which then orchestrates block re-replication while the decommissioning node stops accepting new writes. Monitoring via hdfs dfsadmin -report and the NameNode web UI is essential to track progress and identify bottlenecks.
Key trade-offs include time to decommission (hours to days), elevated network overhead, and rack topology constraints. Following best practices—planning ahead, decommissioning serially, verifying cluster health, and monitoring progress—minimizes operational risk and ensures smooth transitions.