Most HDFS performance complaints are not hardware problems. They are defaults that were chosen when a large cluster was twenty machines and a large file was a few gigabytes, left in place on a cluster three orders of magnitude bigger. The defaults are conservative on purpose -- they have to start safely on a laptop -- and the gap between safe-everywhere and correct-here is where the tuning lives. Three levers do most of the work: the block size, which sets both NameNode memory pressure and job parallelism; the RPC handler pools, which decide whether requests are served or queued; and short-circuit reads, which remove a process hop from every local read. This article covers those three properly and then the second tier -- hedged reads, transfer threads, volume policy, erasure coding -- that matters once the first three are right.
Where the time actually goes
HDFS splits responsibility between two very different services, and a performance problem lives in one or the other. The NameNode holds all metadata in memory and answers every path resolution, block location lookup, create and rename over RPC. It touches no file data at all. The DataNodes hold blocks on local disks and stream bytes to clients.
That split is the first diagnostic question: is the cluster slow because metadata operations are slow, or because data transfer is slow? They have disjoint symptoms and disjoint fixes. Metadata pressure shows up as rising RPC queue time on the NameNode, jobs that take a long time to start, and listing operations that crawl. Data pressure shows up as low aggregate throughput with a healthy NameNode, DataNode threads saturated, or disks at full utilisation.
The measurement to take before changing anything is the NameNode's RPC metrics -- average queue time, average processing time, and call queue length -- exposed over JMX. Queue time far exceeding processing time means requests are waiting for a handler, which is a capacity problem in the handler pool. Processing time itself rising means the NameNode is genuinely busy, which usually means garbage collection or a lock-contended operation such as a huge recursive listing.
Block size — what it really controls
dfs.blocksize defaults to 128 MB and is set per file at creation time. Changing the cluster default does not rewrite existing files; they keep the size they were written with. Any migration to a larger block size is a rewrite, which is why this is a decision to make early.
Block size controls three things at once. It sets the number of blocks, and therefore NameNode memory, since every block is an object in the NameNode's heap. It sets the unit of parallelism for processing engines, because one block typically becomes one input split and therefore one task. And it sets the sequential read efficiency, since larger blocks mean fewer block lookups and fewer connection setups per gigabyte scanned.
Those pull in opposite directions, which is the whole tuning problem. Bigger blocks: less NameNode memory, better scan throughput, fewer tasks -- and coarser parallelism, so a 1 GB block on a small file set can leave most of the cluster idle while a handful of tasks grind, and a single straggler task now covers eight times as much data. Smaller blocks: better parallelism on small datasets, more scheduling overhead, more metadata.
Practical guidance that holds up: 128 MB remains reasonable for mixed clusters with many modest files. 256 MB is the common choice for analytical clusters. 512 MB or 1 GB suits clusters dominated by very large files scanned end to end. Set it per file where the workload varies -- a writer can specify block size at create time, so an ingestion job producing multi-terabyte files can use 1 GB while everything else stays at the default.
One clarification on a claim that circulates in tuning notes: a large block size does not save you from tens of millions of blocks per file -- a petabyte file at 128 MB is about eight million blocks, and no single file is realistically larger. The block-count problem is a cluster-wide total across all files, and it is driven far more by having millions of small files than by the block size of the big ones.
NameNode heap and the small-file problem
Every file, directory and block is an object in the NameNode's heap. A common planning rule of thumb is roughly 150 bytes per object, and while the true figure varies with path length and configuration, the shape of the arithmetic is what matters: ten million files with one block each cost about the same as one million files with ten blocks each, and both are cheap compared with the hundred million small files that a badly configured streaming job can produce in a year.
This is why the small-file problem is a NameNode problem rather than a storage problem. A million 1 MB files occupy a trivial amount of disk and consume the same metadata as a million 1 GB files. They also destroy read performance, because scanning them means a million block lookups and a million connection setups instead of a few thousand sequential reads.
The fixes are all about consolidation. Compact at write time -- have the ingestion job produce files sized near the block size rather than one file per micro-batch, which is the single most effective intervention. Compact after the fact with a periodic rewrite job, which is what every table format's compaction service is doing underneath. Use container formats -- ORC and Parquet with large row groups, sequence files, or HDFS archives -- so many logical records live in one physical file.
When metadata genuinely outgrows one NameNode, the architectural answers are federation, which partitions the namespace across independent NameNodes, usually fronted by router-based federation so clients see one mount table; and a heap large enough to hold the namespace, which brings its own problem -- garbage collection pauses on a hundred-gigabyte heap stall every client in the cluster, so collector choice and pause tuning become first-order concerns rather than afterthoughts.
Handler counts — the RPC thread pools
dfs.namenode.handler.count sets how many threads serve NameNode RPCs. The default of 10 is sized for a demonstration cluster. On a real one, every client operation -- open, create, get block locations, list, rename -- competes for those threads, and when they are all busy, requests queue.
The long-standing sizing guideline is 20 times the natural logarithm of the number of nodes, so a 100-node cluster lands near 92 and a 1,000-node cluster near 138. It is a starting point, not an answer: clusters with many concurrent clients or metadata-heavy workloads want more. The empirical method is better -- raise the count until RPC queue time stops falling, then stop. Beyond that point extra handlers only add contention on the NameNode's internal locks, and enough of them will make things worse.
dfs.datanode.handler.count is the equivalent on the DataNode side and also defaults conservatively. It matters on clusters with many concurrent clients per node, which in practice means colocated HBase or query engines.
The related change that is worth more than the handler count itself on a busy cluster is the service RPC port. By default DataNode heartbeats, block reports and other cluster-internal traffic share one RPC queue with client requests. A burst of client activity therefore delays heartbeats, and delayed heartbeats can make the NameNode start declaring healthy DataNodes dead -- which triggers replication storms that make the load worse. Configuring dfs.namenode.servicerpc-address gives cluster traffic its own port, its own queue and its own handler pool, isolating the two. On any cluster of consequence this is not a tuning nicety, it is a stability measure.
Where a few clients can monopolise the NameNode, the fair call queue adds per-user scheduling to the RPC queue, degrading heavy users' service instead of everyone's. It is the standard answer to one runaway job making the cluster unusable for everybody.
Short-circuit local reads
Normally a client reading a block connects to the DataNode over TCP and the DataNode reads from disk and streams the bytes back. When the client is on the same machine as the data -- which for a well-scheduled job or a colocated HBase RegionServer is the common case -- that hop is pure overhead: a context switch, a copy through the DataNode process, and a loopback socket.
Short-circuit reads remove it. The client asks the DataNode for the block, and the DataNode passes an open file descriptor over a Unix domain socket. The client then reads the file directly. The DataNode still performs the authorisation -- it decides whether to hand over the descriptor -- so this is not a security bypass, but after that the data path involves no other process.
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>Three requirements catch people. The native Hadoop library must be present, since domain-socket file-descriptor passing is not implementable in pure Java; without it the feature silently falls back to normal reads and you conclude it did nothing. The socket path directory must exist with permissions that prevent other users creating it, or the DataNode refuses to start. And both client and DataNode must be configured -- setting it only in the cluster configuration while the client uses its own is a common way to enable a feature that never activates.
The gains are largest for workloads doing many small random reads with good locality -- HBase RegionServers above all, and colocated query engines. They are modest for large sequential scans, where the per-read overhead is already amortised over a lot of data. Verify with the client-side short-circuit read metrics rather than assuming; a configuration that looks right and reports zero short-circuit reads is the usual outcome of the native-library problem above.
Hedged reads and the tail
A single slow disk or a DataNode in a garbage-collection pause turns one read into a multi-second outlier. On a query touching thousands of blocks, that outlier is the query latency, and no average-case tuning helps.
Hedged reads attack it directly: if a read has not returned within a threshold, the client issues a second read against another replica and takes whichever answers first. Two settings control it -- a thread pool size, which enables the feature when non-zero, and a threshold in milliseconds before the hedge fires.
dfs.client.hedged.read.threadpool.size = 20
dfs.client.hedged.read.threshold.millis = 100Set the threshold above your normal read latency -- comfortably above the ninety-fifth percentile -- or every read hedges and you have doubled the cluster's read load to no purpose. Tuned correctly this is one of the highest-value settings available for latency-sensitive readers, and it is measurable: the client exposes counters for how many hedged reads were issued and how many won, and a win rate near zero means the threshold is too low or the problem is not tail latency at all.
The related read-path settings are worth a pass while you are here. Read-ahead on the DataNode tells the operating system to prefetch, which helps sequential scans. Drop-behind hints let large scans avoid evicting the page cache that random readers depend on -- valuable on a node shared between a scan-heavy engine and HBase, where the default behaviour lets the scanner destroy the other workload's cache.
DataNode I/O configuration
Disks are individual, not a RAID set. HDFS wants a JBOD layout with each disk listed separately in dfs.datanode.data.dir, because it does its own replication and gets more parallelism from independent spindles than from a striped volume. RAID underneath HDFS costs capacity and throughput to solve a problem HDFS already solved.
Volume choosing policy decides which of those disks a new block lands on. Round-robin is the default and is fine when disks are uniform; the available-space policy biases towards emptier disks and is what you want after adding capacity to existing nodes, since round-robin will otherwise leave new disks under-filled indefinitely. The disk balancer redistributes existing blocks across volumes within a node and is a separate tool from the cluster balancer, which moves data between nodes -- they solve different imbalances and people routinely run the wrong one.
Transfer threads -- dfs.datanode.max.transfer.threads, historically the xceiver limit -- caps concurrent block transfers per DataNode. The modern default of 4096 is adequate for most workloads and is a well-known thing to raise for HBase, whose many open store files and concurrent readers can exhaust it. Hitting the limit produces errors about exceeding the xceiver count in DataNode logs, which is one of the more legible failure messages in Hadoop.
Balancer bandwidth throttles how fast the cluster balancer moves data. The default is deliberately timid, so a balance after adding a rack can take weeks; raising it shortens that at the cost of network contention with real work. Raise it during quiet periods and put it back.
Replication, rack awareness and erasure coding
Three-way replication is the default and it costs 200 percent overhead. It buys durability, read parallelism -- three replicas means three candidate sources, which is what makes hedged reads and locality-aware scheduling possible -- and fast recovery, since re-replicating a lost block is a copy.
Rack awareness is what makes that placement sensible: with a topology script configured, HDFS places one replica locally, one on a remote rack and one on the same remote rack, tolerating a rack failure while keeping most write traffic off the cross-rack links. A cluster without a topology script has every node in one default rack, which means the placement policy is doing nothing for you and a single rack failure can lose data. Check this on any cluster you inherit; it is silently wrong more often than it should be.
Erasure coding changes the arithmetic. With a Reed-Solomon scheme such as six data cells and three parity cells, storage overhead falls from 200 percent to 50 percent for the same tolerance of three failures. The costs are real: writes compute parity and spread cells across nine nodes, so there is no single local replica and therefore no data locality and no short-circuit read; recovering a lost cell means reading six others and reconstructing, which is far more expensive than copying a replica; and the striped layout means small files waste space and gain nothing.
The resulting rule is straightforward. Erasure-code cold, large, infrequently read data -- archives, old partitions -- and keep replication for hot data, small files and anything whose readers depend on locality. Because the policy is set per directory, both can coexist in one cluster, and an age-based migration between them is the standard arrangement.
The NameNode's own bottlenecks
Beyond handler counts, three things determine NameNode responsiveness.
Garbage collection. The namespace lives in one heap and that heap gets very large. A stop-the-world pause blocks every metadata operation cluster-wide, so long pauses look to users like a total outage. Use a low-pause collector, size the heap with headroom rather than to the edge, and monitor pause duration as a first-class metric -- it is frequently the real cause behind 'the cluster froze for thirty seconds'.
Edit log latency. Every namespace mutation is written to the edit log, which in a highly-available setup means a quorum of JournalNodes must acknowledge before the operation returns. JournalNode disk latency therefore sits directly in the path of every create, rename and delete. Put journal directories on fast, dedicated storage; a JournalNode sharing a spindle with something else is a cluster-wide write latency problem waiting to happen.
Lock contention. The namespace is protected by a global lock, and some operations hold it for a long time -- a recursive listing or delete over a directory with millions of entries is the classic offender, and it stalls every other client while it runs. This is a workload problem more than a configuration one: teach jobs not to list enormous directories, and prefer many shallow directories to one flat directory with a million children.
For read-heavy metadata workloads, the observer NameNode serves consistent reads from a standby, offloading listing and location lookups from the active one. Clients coordinate with a state-synchronisation call to avoid reading stale metadata, so it is consistent rather than eventually consistent -- but it does add a round trip, which is why it pays off for read-dominated workloads and not for write-heavy ones.
A method, not a list of settings
Measure first, and measure the right layer. Establish whether the problem is metadata or data. NameNode JMX gives RPC queue time, processing time, call queue length and GC pauses; DataNode metrics give transfer thread counts and volume latency; client-side counters report short-circuit and hedged read behaviour. A synthetic benchmark such as the standard distributed I/O test gives a throughput baseline to compare against after a change.
Change one thing and re-measure. Tuning sessions that adjust six settings at once produce a cluster nobody can reason about, and the usual outcome is that one change helped, one hurt, and the net was a wash.
Fix the workload before the configuration. The biggest wins available in most clusters are not settings at all: stop producing small files, stop listing million-entry directories, write files sized to the block size, and use columnar formats with sensible row group sizes. No handler count compensates for a job that creates fifty thousand files an hour.
Know the anti-patterns. RAID under DataNodes. A cluster with no rack topology script. Handler counts raised until the NameNode is contending with itself. Hedged reads with a threshold below normal latency, doubling read load. Erasure coding applied to hot data or small files. Short-circuit reads configured without the native library, delivering nothing. Each of these is common, each looks like tuning, and each costs performance rather than gaining it.