Why architecture matters here
Storage in HDFS is only as available as its most-full disk. A DataNode does not stripe a block across its volumes; each block replica lives entirely on one disk. When a disk fills, writes that the volume-choosing policy routes to it fail, and depending on configuration a single failed volume can take the whole DataNode out of service. So a node that reports 60% average utilization but has one disk at 99% is not 40% away from trouble — it is one unlucky block placement away from a write failure, and the average hides exactly the disk that is about to break. Intra-node balance is therefore a reliability property, not a tidiness preference.
The imbalance is architecturally inevitable because block placement and disk provisioning happen on completely different clocks. Disks get added, replaced, and reformatted over a node's multi-year life; blocks get placed continuously by a policy that only sees the current write. The default round-robin volume policy spreads new blocks evenly by count, which means a freshly added empty disk gets the same number of new blocks as the near-full veterans — so it never catches up. The available-space policy biases writes toward emptier disks, but it over-corrects: a new disk becomes the write magnet for the whole node and its spindle saturates while the others idle. Either way, the steady state is skew, and nothing in the normal write path ever moves already-written data to fix it.
Before the Disk Balancer existed (it arrived in Hadoop 3.0 via HDFS-1312), operators' only remedy for a hot disk was crude and dangerous: decommission the DataNode, physically rebalance or reformat, and let re-replication refill it — hours of reduced redundancy and cluster-wide network traffic to fix a problem local to one machine. Or they would manually move block files and their meta companions between volume directories, a hand operation that corrupts the DataNode's block map if done while the process is running. The Disk Balancer turns this into a first-class, online, throttled operation that the DataNode itself performs safely, updating its in-memory volume map as it goes.
The architectural reason it is a separate tool from the cluster Balancer is that the two operate at different layers and must not be confused. The cluster Balancer talks to the NameNode, reasons about replica placement policy and rack awareness, and shuffles blocks over the network between nodes — an expensive, cluster-global operation constrained by the need to preserve fault-tolerance invariants. The Disk Balancer never touches the network or the NameNode's placement logic; it moves bytes between local disks on one host, so it is cheap, safe to run frequently, and invisible to the rest of the cluster. Understanding that split is what stops teams from reaching for a decommission when a five-minute local move would have done it.
The architecture: every piece explained
Top row: discovery and planning. The unit of work is a single DataNode with N physical volumes (disks). The Disk Balancer first gathers a volume report — used and free bytes per disk, plus the DataNode's storage types (DISK, SSD, ARCHIVE), because it only balances within a storage type. From that report the plan generator computes, for this one node, a set of source-to-destination moves: which over-full disk should shed how many bytes to which under-full disk to bring every volume within a configured threshold of the node's average. The output is a plan — a JSON document, one per node, that you inspect and then submit for execution. Planning and execution are deliberately separate so an operator can review the intended moves before any bytes move.
Middle row: execution mechanics. The move executor runs inside the DataNode and copies block files (and their checksum meta files) from the source volume to the destination volume. Because this competes with real client and replication I/O, every move is governed by a bandwidth throttle — a megabytes-per-second cap so balancing never starves the workload the node exists to serve. After each block is copied, the DataNode updates its block metadata — the in-memory FsVolume map that records which volume holds each replica — so reads immediately find the block on its new disk. Only after the copy is confirmed does the executor verify the checksum at the destination and then delete the source replica, making the move atomic from the reader's perspective: at no point is the block missing.
Bottom-left: the policy that shapes the future. The volume choosing policy — round-robin (even by block count) versus available-space (biased toward emptier disks, with a tunable balanced-space threshold and preference fraction) — decides where new writes land. The Disk Balancer fixes the existing skew; the volume policy determines whether the skew comes back. The two are complementary: rebalance once, then set an available-space policy so the disks stay level. Bottom-right: metrics expose planned-bytes versus moved-bytes so you can watch progress and confirm a plan actually ran to completion rather than stalling part way.
Bottom strip: the operational surface. The Disk Balancer is driven by hdfs diskbalancer subcommands — -plan to generate, -execute to run, -query to check status, and -cancel to stop — and gated by dfs.disk.balancer.enabled. The three knobs that matter operationally are the threshold (how close to the node average each disk must be before the balancer leaves it alone), the bandwidth cap (how aggressively moves compete with foreground I/O), and the ability to query and cancel a running plan so a rebalance can be paused the instant the node gets busy.
End-to-end flow
Follow a real node. dn-42 has twelve 8 TB disks; ten were installed at cluster build, two were added six months later to grow capacity. Under the default round-robin volume policy the two new disks each hold roughly a third of what the veterans hold, so /data/1 through /data/10 sit near 88% while /data/11 and /data/12 idle at 30%. Nothing is wrong at the cluster level — the node reports about 78% average — but overnight batch jobs writing large blocks keep routing a proportional share to the near-full veterans, and the on-call gets a 'volume nearly full' warning for /data/3.
The operator runs hdfs diskbalancer -plan dn-42. The tool reads the node's volume report, sees the spread, and with a 10% threshold computes a plan: move roughly 4.6 TB total, shedding bytes from the ten hot disks onto the two cold ones until every volume lands within 10% of the ~78% average. The plan is written as JSON to HDFS; the operator inspects it, confirms the source and destination volumes look right, and no data has moved yet. This inspect-before-execute gate is deliberate — a plan is a proposal, not an action.
Execution: hdfs diskbalancer -execute /plans/dn-42.plan.json hands the plan to the DataNode. The move executor begins copying blocks from /data/3 to /data/11, throttled to the configured cap (say 10 MB/s per move so it never eats more than a sliver of the node's disk bandwidth). For each block it copies the block file and its .meta checksum companion, verifies the checksum at the destination, updates the FsVolume map so the replica is now recorded on /data/11, and only then deletes the copy on /data/3. A concurrent client reading that block during the window is unaffected: the old copy stays live until the new one is verified, so there is never a moment where the replica is absent from the node.
Progress and control: the operator polls hdfs diskbalancer -query dn-42 and watches moved-bytes climb toward the planned 4.6 TB. Midway, a large Spark job lands on the cluster and the node's foreground I/O spikes; because balancing is throttled it yields gracefully, but to be safe the operator could -cancel the plan and resume it in the next maintenance window — the already-moved blocks stay where they are, no work is lost. When the plan completes, all twelve disks sit near 78% ± a few percent. Finally the operator addresses the root cause: switching the node's dfs.datanode.fsdataset.volume.choosing.policy to the available-space policy so future writes prefer the emptier disks and the node self-levels instead of drifting back into skew.
The subtle correctness detail that makes all of this safe is the copy-verify-delete ordering, and it is worth dwelling on because it is exactly where naive manual rebalancing goes wrong. When an operator hand-moves block files with mv, there is a window where the DataNode's in-memory volume map still points at the old path while the file is gone — a read in that window sees a missing replica and the NameNode may trigger unnecessary re-replication, or worse the block is briefly corrupt from the cluster's point of view. The Disk Balancer closes that window by construction: the source replica is authoritative until the destination copy exists and passes a checksum comparison, the volume map flip is a single in-memory update, and only after that does the source get unlinked. The DataNode is a live participant that keeps its own bookkeeping consistent at every step, which is why the move can happen online with clients actively reading and writing the node.