Why it matters
Even in a cloud-first world, on-premise HDFS clusters still hold a very large share of enterprise analytical data. Regulated industries like banking, healthcare, and telecommunications keep customer records inside their own datacenters for compliance and sovereignty reasons. Manufacturing companies keep sensor archives on-premise because moving them to the cloud costs more in bandwidth than the analytical value returns. Any engineer working on Spark, Hive, HBase, Trino, or Impala will encounter HDFS regularly and needs to understand its guarantees and quirks.
Beyond the on-premise story, HDFS taught the industry how to think about large-scale distributed storage. The idea of a metadata master coordinating a large fleet of dumb block servers shows up in Amazon S3, Google Colossus, Azure Blob Storage, and every other planet-scale storage system. Understanding HDFS is understanding the pattern.
The architecture
HDFS has two kinds of processes. The NameNode is a single master process that holds the entire filesystem namespace in memory: the directory tree, file-to-block mapping, and the current location of every block replica across the cluster. It never touches actual file data. The DataNodes are workers, one per storage server, that own local disks and serve raw block reads and writes. There is no fancy protocol between DataNodes and the NameNode; DataNodes simply heartbeat every three seconds and send a block report roughly once an hour telling the NameNode which blocks they hold.
Files in HDFS are split into fixed-size blocks, typically 128 megabytes, and each block is replicated three times by default. Replicas are placed with rack awareness: one replica on the local rack, two on a different rack, so that a rack failure never loses a block. Clients read directly from DataNodes without proxying through the NameNode, which is why HDFS can serve terabyte-per-second aggregate throughput from a single-master architecture. The NameNode is on the metadata path but not on the data path.
How it works end to end
When a client writes a file, it first asks the NameNode to allocate a new file entry and to pick three DataNodes for the first block. The NameNode returns those three targets, and the client opens a pipeline: it sends bytes to the first DataNode, which forwards them to the second, which forwards them to the third. Each DataNode writes to its local disk as bytes arrive. Once the block is fully written and every DataNode has acknowledged, the client asks the NameNode for another three targets for the next block, and so on until the file is complete. This pipeline design keeps write throughput close to network line speed.
Reading is even simpler. The client asks the NameNode for the list of blocks and their DataNode locations for a given path. It then opens direct sockets to the DataNodes in parallel and streams block data back. The NameNode is not involved once the client has the block map, so a single NameNode can support tens of thousands of concurrent readers as long as it has enough heap for its in-memory metadata structures.
Deletes and renames are metadata-only operations that happen entirely inside the NameNode. Data on DataNodes is reclaimed lazily via a background garbage collection cycle. This is why deletes appear instant even for terabyte files.