All 40 articles, sorted alphabetically
DistCp
How DistCp uses a MapReduce job to parallelize large HDFS-to-HDFS or HDFS-to-cloud copies. Covers throughput tuning, incremental sync, and gotchas for…
Read article →HDFS Balancer architecture
Deep-dive on the HDFS Balancer and Disk Balancer: threshold policy, source/proxy/target replica moves, placement invariants, bandwidth governors, iter…
Read article →YARN Capacity Scheduler
Deep-dive on the YARN Capacity Scheduler: queue hierarchy and capacity guarantees, elastic borrowing and max capacities, preemption to reclaim, user l…
Read article →HDFS NameNode checkpointing architecture
Deep-dive on HDFS NameNode checkpointing: the fsimage snapshot and append-only edit log, JournalNode quorum durability in HA, the Standby/Secondary ch…
Read article →Hadoop Node Decommissioning
Graceful DataNode removal. Block re-replication.
Read article →HDFS Disk Balancer architecture
Deep-dive on the HDFS Disk Balancer: why disks within one DataNode drift out of balance, plan generation, throttled copy-verify-delete block moves, th…
Read article →DistCp architecture
Deep-dive on Hadoop DistCp: copy listings and dynamic split strategies, snapshot-diff incremental replication, bandwidth throttling for shared WAN lin…
Read article →HDFS centralized cache management architecture
Deep-dive on HDFS centralized cache management: cache pools and directives with quotas, the NameNode cache manager driving DataNodes to mmap/mlock blo…
Read article →HDFS Federation architecture
Deep-dive on HDFS Federation: multiple independent NameNodes with per-namespace block pools over shared DataNodes, ViewFS client mount tables vs Route…
Read article →HDFS quotas architecture
Deep-dive on HDFS quotas: namespace (object-count) quotas, space (replicated-byte) quotas, storage-type quotas, how QuotaCounts are cached on director…
Read article →HDFS Router-Based Federation (RBF): Architecture Deep-Dive
How HDFS Router-Based Federation scales past the single-NameNode ceiling with stateless routers, a State Store mount table, and one global namespace o…
Read article →HDFS snapshots architecture
Deep-dive on HDFS snapshots: instant, read-only, copy-on-write captures of a directory subtree that share immutable blocks with the live tree and cost…
Read article →The HDFS write pipeline
Deep-dive on the HDFS write path: DFSOutputStream's data and ack queues, rack-aware pipeline construction, BlockReceiver rela…
Read article →Hadoop HDFS and YARN Architecture in Depth
A 2500-word walkthrough of Hadoop: HDFS NameNode/Standby, DataNodes, YARN ResourceManager, NodeManagers, ApplicationMaster, schedulers, and security.
Read article →Hadoop Kerberos
Why every production Hadoop cluster runs Kerberos, how principals, keytabs, and tickets work, and the operational patterns that keep Kerberized cluste…
Read article →HDFS NameNode HA architecture
Deep-dive on HDFS NameNode High Availability: Quorum Journal Manager and epoch fencing, hot standby via edit tailing, ZKFC and ZooKeeper election, fen…
Read article →Hadoop Rack Awareness
Topology script maps host → rack. Placement uses it.
Read article →Ranger + Hadoop architecture
Deep-dive on Ranger authorization for Hadoop: admin, policies, tags, plugins (HDFS/Hive/HBase), audit sink, Kerberos, masking.
Read article →HDFS short-circuit reads -- bypassing the DataNode for local data
Deep-dive on HDFS short-circuit reads: the normal DataNode read path, local reads with co-located compute, short-circuit reading the block file direct…
Read article →External shuffle service -- decoupling shuffle from executors
Deep-dive on the external shuffle service: the shuffle problem and executor-local fragility, per-node shuffle service decoupling data from executor li…
Read article →Speculative execution -- racing around stragglers
Deep-dive on speculative execution: the straggler problem (a job finishes only when all tasks finish), straggler detection, launching a speculative du…
Read article →YARN Fair Scheduler architecture
Deep-dive on the YARN Fair Scheduler: the weighted queue hierarchy, the fair-share computation that tracks live demand, the assign-to-most-below-share…
Read article →YARN node labels architecture
Deep-dive on YARN node labels: mapping nodes into named partitions so the scheduler places GPU, high-memory, licensed, and spot workloads on the right…
Read article →HDFS Blocks and Replication
Deep dive on HDFS block size, 3-way replication, rack-aware placement, and why the classic policy of 'one local plus two on a rem…
Read article →HDFS Checkpoints and JournalNodes
How HDFS combines periodic FsImage checkpoints with a JournalNode-quorum edit log to give the NameNode durability, fast startup, and HA-compatible sta…
Read article →HDFS DataNode Deep Dive
How the DataNode stores blocks on local disks, heartbeats to the NameNode, participates in write pipelines, and self-scans for corruption. Covers JBOD…
Read article →HDFS Encryption Zones
How HDFS Encryption Zones give you per-file AES encryption with client-side key handling and KMS-managed key wrapping. Covers key hierarchy, performan…
Read article →HDFS Erasure Coding
How Reed-Solomon erasure coding replaces three-way replication for cold data. Covers the RS(6,3) scheme, striping, reconstruction IO cost, and when to…
Read article →HDFS Federation
How HDFS Federation splits the namespace across multiple NameNodes to break the single-master heap bottleneck. Covers block pools, router-based federa…
Read article →HDFS High Availability
How HDFS achieves NameNode HA with an active/standby pair, a JournalNode quorum for the shared edit log, ZooKeeper for leader election, and fencing to…
Read article →HDFS NameNode Deep Dive
How the NameNode manages the HDFS namespace, why the edit log matters, how high availability with JournalNodes works, and how to size heap for hundred…
Read article →HDFS Overview
Complete walkthrough of HDFS: NameNode metadata, DataNode blocks, replication, client read/write paths, and why HDFS is optimized for large files and …
Read article →HDFS Permissions and ACLs
How HDFS enforces access control using POSIX permission bits plus extended ACLs, and how Ranger centralizes policy across HDFS, Hive, HBase, and other…
Read article →HDFS Safe Mode
Why the NameNode boots into safe mode, what block-report threshold triggers the exit, and when operators should force-exit it — with the risks that co…
Read article →HDFS Small Files Problem
Why HDFS is fundamentally optimized for large files, how millions of small files exhaust NameNode heap, and the standard techniques to consolidate the…
Read article →HDFS Performance Tuning
The three highest-impact HDFS tuning levers: block size selection, NameNode and DataNode handler thread pools, and short-circuit local reads.
Read article →WebHDFS and HttpFS
How WebHDFS and HttpFS expose HDFS over HTTP so non-JVM clients can read and write files. Covers the two deployment styles, security via Kerberos SPNE…
Read article →YARN ApplicationMaster
How the ApplicationMaster negotiates containers, tracks task progress, handles failures, and shuts down cleanly. Covers the AM protocols and design pa…
Read article →YARN cgroups Container Isolation
Linux cgroups for CPU + memory limits.
Read article →YARN Preemption
Kill containers to give resources to underserved queues.
Read article →