All 117 articles, sorted alphabetically
Apache Ambari
Cluster management + monitoring. Now Attic.
Read article →Airflow on Hadoop
Modern DAG orchestrator. Replaces Oozie.
Read article →Apache Atlas
Data governance for Hadoop. Track lineage.
Read article →Apache Knox
REST proxy for Hadoop services. Single access point.
Read article →Apache Kudu
Columnar storage for fast analytics + writes.
Read article →Apache NiFi
Visual dataflow programming. GUI-driven pipeline.
Read article →Apache Ozone
HDFS successor for billions of small objects.
Read article →Apache Ranger
Centralized fine-grained access control.
Read article →Cloud Object Storage as HDFS Replacement
S3/GCS/ABS as data lake. HDFS optional.
Read article →Cloudera Manager
Commercial Hadoop cluster mgmt. Successor to Ambari.
Read article →GCP Dataproc
Fast startup. Cost-optimized. Serverless too.
Read article →Delta Lake
ACID on Parquet. Databricks native.
Read article →DistCp
How DistCp uses a MapReduce job to parallelize large HDFS-to-HDFS or HDFS-to-cloud copies. Covers throughput tuning, incremental sync, and gotchas for…
Read article →DistCp Advanced Patterns
Cross-cluster + cloud migration + delta.
Read article →AWS EMR
Ephemeral clusters. S3 native. Spark + Hive + more.
Read article →Apache Flume
Streaming log collection. Kafka mostly replaces it.
Read article →Hadoop 3 Key Features
Erasure coding, YARN v2, Docker containers, EC.
Read article →Apache Ambari
How Ambari centralizes installation, configuration, service lifecycle, and monitoring for Hadoop clusters. Covers blueprints, alerts, and the shift to…
Read article →Hadoop Backup + DR
Snapshots + DistCp to backup cluster or cloud.
Read article →HDFS Balancer architecture
Deep-dive on the HDFS Balancer and Disk Balancer: threshold policy, source/proxy/target replica moves, placement invariants, bandwidth governors, iter…
Read article →Hadoop Benchmarks
TestDFSIO, TeraSort, TPC-DS-Impala.
Read article →Hadoop Capacity Planning
How to plan Hadoop cluster capacity across four dimensions (storage, compute, memory, network), where headroom actually matters, and when to expand ve…
Read article →YARN Capacity Scheduler
Deep-dive on the YARN Capacity Scheduler: queue hierarchy and capacity guarantees, elastic borrowing and max capacities, preemption to reclaim, user l…
Read article →Hadoop Cloud Migration Playbook
Inventory + assess + phased migration.
Read article →Hadoop Cluster Planning
Sizing + hardware + network for a fresh cluster.
Read article →Hadoop Cost Management
Storage tiering + compression + old data cleanup.
Read article →Why Hadoop Is Declining
Cloud + object storage + Databricks + Snowflake ate the lunch.
Read article →Hadoop DataNode Decommission
The safe decommissioning workflow: excludes file, refreshNodes, block re-replication, and final removal. When decommission stalls and how to unstick i…
Read article →Hadoop Node Decommissioning
Graceful DataNode removal. Block re-replication.
Read article →DistCp architecture
Deep-dive on Hadoop DistCp: copy listings and dynamic split strategies, snapshot-diff incremental replication, bandwidth throttling for shared WAN lin…
Read article →Druid architecture
Deep-dive on Apache Druid: ingest, segments, historicals, brokers, coordinator, rollups, bitmap indexes, deep storage.
Read article →Hadoop Ecosystem in 2026
Where each tool stands + successors.
Read article →HDFS erasure coding architecture
Deep-dive on HDFS erasure coding: EC policies, block groups, reads, reconstruction, migration, small-file impact, and ops.
Read article →HDFS Federation architecture
Deep-dive on HDFS Federation: multiple independent NameNodes with per-namespace block pools over shared DataNodes, ViewFS client mount tables vs Route…
Read article →Hadoop HDFS and YARN Architecture in Depth
A 2500-word walkthrough of Hadoop: HDFS NameNode/Standby, DataNodes, YARN ResourceManager, NodeManagers, ApplicationMaster, schedulers, and security.
Read article →Hive architecture
Deep-dive on Apache Hive: HS2, Metastore, Tez execution, LLAP daemons, ACID tables, formats, Ranger security, partitioning, CBO.
Read article →Impala architecture
Deep-dive on Apache Impala: frontend planner, backend executor, LLVM codegen, statestore, catalogd, admission control, formats.
Read article →JVM Tuning for Hadoop Services
G1GC + heap sizing per role.
Read article →Hadoop Kerberos
Why every production Hadoop cluster runs Kerberos, how principals, keytabs, and tickets work, and the operational patterns that keep Kerberized cluste…
Read article →Kudu architecture
Deep-dive on Apache Kudu: tables, tablets, Raft replication, columnar DRS + deltas, upsert, Impala + Spark integration.
Read article →Hadoop Log Aggregation
Ship container logs to HDFS + query via Timeline.
Read article →Hadoop Migration to Cloud
Playbook: HDFS to S3, Hadoop to EMR/Dataproc.
Read article →Hadoop Monitoring
JMX metrics + Prometheus/Grafana or CM/Ambari.
Read article →HDFS NameNode HA architecture
Deep-dive on HDFS NameNode High Availability: Quorum Journal Manager and epoch fencing, hot standby via edit tailing, ZKFC and ZooKeeper election, fen…
Read article →Ozone architecture
Deep-dive on Apache Ozone: Ozone Manager, SCM, storage containers, Ratis consensus, DataNodes, replication, S3 gateway.
Read article →Hadoop Rack Awareness
Topology script maps host → rack. Placement uses it.
Read article →Apache Ranger
How Ranger centralizes access policies across HDFS, Hive, HBase, Kafka, and Solr — with tag-based rules, row/column-level security, and comprehensive …
Read article →Ranger + Hadoop architecture
Deep-dive on Ranger authorization for Hadoop: admin, policies, tags, plugins (HDFS/Hive/HBase), audit sink, Kerberos, masking.
Read article →Hadoop Rolling Upgrade
How to perform a rolling upgrade of a Hadoop cluster: NameNode HA sequence, DataNode batch upgrade, service-by-service order, and the finalize step.
Read article →Hadoop Security Best Practices
Kerberos + Ranger + Atlas + KMS + TLS.
Read article →HDFS short-circuit reads -- bypassing the DataNode for local data
Deep-dive on HDFS short-circuit reads: the normal DataNode read path, local reads with co-located compute, short-circuit reading the block file direct…
Read article →External shuffle service -- decoupling shuffle from executors
Deep-dive on the external shuffle service: the shuffle problem and executor-local fragility, per-node shuffle service decoupling data from executor li…
Read article →Speculative execution -- racing around stragglers
Deep-dive on speculative execution: the straggler problem (a job finishes only when all tasks finish), straggler detection, launching a speculative du…
Read article →Hadoop Upgrade Patterns
Rolling upgrades. HA prerequisite.
Read article →When Hadoop Still Makes Sense
On-prem regulated data + existing investment.
Read article →YARN placement architecture
Deep-dive on YARN placement: labels, constraints, gang scheduling, preemption, and reservations under a queue hierarchy.
Read article →ZooKeeper in Hadoop
How ZooKeeper underpins HDFS NameNode HA, YARN ResourceManager HA, HBase region assignment, and every other Hadoop coordination need.
Read article →HDFS Balancer
How the HDFS Balancer moves blocks from overused DataNodes to underused ones to keep storage utilization balanced. Covers the algorithm, bandwidth thr…
Read article →HDFS Blocks and Replication
Deep dive on HDFS block size, 3-way replication, rack-aware placement, and why the classic policy of 'one local plus two on a rem…
Read article →HDFS Checkpoints and JournalNodes
How HDFS combines periodic FsImage checkpoints with a JournalNode-quorum edit log to give the NameNode durability, fast startup, and HA-compatible sta…
Read article →HDFS CLI Deep Dive
The two HDFS commands every operator lives in — hdfs dfs and hdfs dfsadmin — with practical recipes for inspection, data movement, permissions, and ad…
Read article →HDFS DataNode Deep Dive
How the DataNode stores blocks on local disks, heartbeats to the NameNode, participates in write pipelines, and self-scans for corruption. Covers JBOD…
Read article →HDFS Encryption Zones
How HDFS Encryption Zones give you per-file AES encryption with client-side key handling and KMS-managed key wrapping. Covers key hierarchy, performan…
Read article →HDFS Erasure Coding
How Reed-Solomon erasure coding replaces three-way replication for cold data. Covers the RS(6,3) scheme, striping, reconstruction IO cost, and when to…
Read article →HDFS Federation
How HDFS Federation splits the namespace across multiple NameNodes to break the single-master heap bottleneck. Covers block pools, router-based federa…
Read article →HDFS High Availability
How HDFS achieves NameNode HA with an active/standby pair, a JournalNode quorum for the shared edit log, ZooKeeper for leader election, and fencing to…
Read article →HDFS NameNode Deep Dive
How the NameNode manages the HDFS namespace, why the edit log matters, how high availability with JournalNodes works, and how to size heap for hundred…
Read article →HDFS Overview
Complete walkthrough of HDFS: NameNode metadata, DataNode blocks, replication, client read/write paths, and why HDFS is optimized for large files and …
Read article →HDFS Permissions and ACLs
How HDFS enforces access control using POSIX permission bits plus extended ACLs, and how Ranger centralizes policy across HDFS, Hive, HBase, and other…
Read article →HDFS Quotas
How HDFS namespace and space quotas work at the NameNode inode level, why per-user or per-team quotas are essential in multi-tenant clusters, and how …
Read article →HDFS Safe Mode
Why the NameNode boots into safe mode, what block-report threshold triggers the exit, and when operators should force-exit it — with the risks that co…
Read article →HDFS Small Files Problem
Why HDFS is fundamentally optimized for large files, how millions of small files exhaust NameNode heap, and the standard techniques to consolidate the…
Read article →HDFS Snapshots
How HDFS snapshots capture a read-only view of a directory in constant time using copy-on-write semantics. Covers use cases (backup, rollback, audit),…
Read article →HDFS Trash
How HDFS trash rewrites -rm into a move, when checkpoints are rolled, and how to size fs.trash.interval to balance user safety against storage cost.
Read article →HDFS Performance Tuning
The three highest-impact HDFS tuning levers: block size selection, NameNode and DataNode handler thread pools, and short-circuit local reads.
Read article →Azure HDInsight
Managed Hadoop on Azure. Legacy — Fabric supplants.
Read article →Iceberg + Trino
Table format + query engine replacing Hive.
Read article →Kerberos Deployment for Hadoop
KDC + realms + principals. Enterprise auth foundation.
Read article →Kerberos on Hadoop
Strong auth for Hadoop services. KDC-based tickets.
Read article →MapReduce Combiner
How combiners run map-side to pre-aggregate values, why they can cut shuffle bytes by 100x or more, and when they are unsafe to use.
Read article →MR Distributed Cache
Ship files/jars to all tasks.
Read article →MapReduce Is Dead
Spark + Flink + Trino displaced MR.
Read article →MapReduce Map Phase
How the map phase reads HDFS blocks, converts bytes to records, applies the mapper, and manages the circular buffer that spills to disk. The mechanics…
Read article →Mapper
Convert input records to key-value pairs.
Read article →MR Output Committer
Atomically move task output to final location.
Read article →MapReduce Overview
How MapReduce turns a distributed problem into map, shuffle, and reduce phases. The programming model, the runtime, and why it still matters even in t…
Read article →MapReduce Partitioner
How partitioners route intermediate keys to reducers, why HashPartitioner is the default, and when to use TotalOrderPartitioner or custom partitioners…
Read article →MapReduce Reduce Phase
How the reduce phase iterates over sorted key groups, applies the reduce function, and writes final output to HDFS through the OutputFormat.
Read article →Reducer
Receives keyed groups, produces final output.
Read article →MapReduce Shuffle and Sort
How reducers fetch their partitions from every mapper's local disk, how merge-sort assembles the reduce input, and why shuffle is…
Read article →MapReduce Speculative Execution
How MapReduce launches speculative copies of slow tasks, why it wastes some CPU to save far more wall-clock time, and when to disable it.
Read article →MapReduce Tuning
Parallelism, memory, shuffle, compression.
Read article →Modern Lakehouse Architecture
Object storage + open table format + query engine.
Read article →Apache Oozie
XML-defined workflow orchestrator. Airflow eats its lunch.
Read article →Ozone vs HDFS
Object store for scale beyond HDFS.
Read article →Presto/Trino vs Hive
Interactive SQL. Faster than Hive.
Read article →Ranger Policy Deep Dive
Resource-based + tag-based policies.
Read article →Apache Sqoop
Bulk transfer relational data. Deprecated but still used.
Read article →WebHDFS and HttpFS
How WebHDFS and HttpFS expose HDFS over HTTP so non-JVM clients can read and write files. Covers the two deployment styles, security via Kerberos SPNE…
Read article →YARN ApplicationMaster
How the ApplicationMaster negotiates containers, tracks task progress, handles failures, and shuts down cleanly. Covers the AM protocols and design pa…
Read article →YARN Capacity Scheduler
How the Capacity Scheduler organizes cluster capacity into a tree of queues with minimum guarantees and elastic maximums, and how borrowing works betw…
Read article →YARN cgroups Container Isolation
Linux cgroups for CPU + memory limits.
Read article →YARN Containers
What a YARN container really is, how memory and vcores are enforced by cgroups, and why the abstraction lets one cluster run wildly different workload…
Read article →YARN Docker Container Support
Run tasks in Docker containers.
Read article →YARN Fair Scheduler
How Fair Scheduler distributes resources equally among active users and queues, and when it beats Capacity Scheduler.
Read article →YARN Node Labels and Placement
How YARN node labels let you segregate GPU, SSD, or high-memory nodes into pools that only labeled applications can use.
Read article →YARN Node Labels
Tag nodes; steer jobs to specific hardware.
Read article →YARN NodeManager
How the NodeManager launches containers, enforces resource limits via cgroups, localizes job resources, and reports health to the ResourceManager.
Read article →YARN Opportunistic Containers
Launch above guaranteed capacity; preempt if needed.
Read article →YARN Overview
How YARN separates resource management from application logic, why it replaced MapReduce v1's JobTracker, and the four core compo…
Read article →YARN Preemption
Kill containers to give resources to underserved queues.
Read article →YARN Queues Deep Dive
Named allocation pools. Hierarchical + per-user limits.
Read article →YARN Queue Hierarchies
How to design YARN queue hierarchies that scale organizationally: leaf vs parent queues, ACLs, submission policies, and per-user limits.
Read article →YARN ResourceManager
The internal structure of the YARN ResourceManager: the scheduler, the applications manager, queue configuration, and the ZooKeeper-based HA that keep…
Read article →YARN Resource Types (GPU, FPGA)
Custom resource types beyond memory + CPU.
Read article →YARN Timeline Server
Application history storage for post-mortem analysis.
Read article →ZooKeeper's Role in Hadoop
Coordination service. HA elections. HBase requires.
Read article →