Why it matters
Understanding MapReduce is understanding the assumptions every subsequent framework inherited. Spark's stages are lineage-tracked mini MapReduces. Flink's operators can be modeled as long-lived map or reduce operations. The vocabulary of partitioner, combiner, and shuffle came from here. Debugging Spark performance issues frequently requires MapReduce-level intuitions about shuffle cost, partitioner choice, and skew.
Even in 2026, plenty of nightly ETL runs as MapReduce because it is simple, stable, and produces predictable results. Familiarity with the model is table stakes for anyone owning a Hadoop cluster.
The architecture
A MapReduce job has three main phases and several supporting components. The input is split into equal-size chunks (splits), typically one per HDFS block, and each split feeds one map task. Map tasks emit intermediate key-value pairs to a local buffer that spills to disk when full.
Between map and reduce, the shuffle phase moves intermediate pairs across the network to reducers, grouping by key. Reduce tasks consume the sorted key groups and produce output written back to HDFS. Optional combiners, partitioners, and speculative execution all customize this pipeline.
How it works end to end
Job submission goes through the YARN ApplicationMaster (specifically the MRAppMaster). The AM computes input splits from the InputFormat, requests one map container per split, and launches map tasks. Each map task reads its split from HDFS (usually locally, thanks to locality scheduling), applies the user's map function, and writes intermediate output to local disk sorted by partitioner.
Once map tasks begin completing, reducers start fetching their partitions from map outputs. A reducer with partition ID 3 pulls only the records assigned to partition 3 from every map task. These fetched partitions are merge-sorted locally, then the user's reduce function is applied to each key group.
Reducer output is written to HDFS, one file per reducer. The MRAppMaster tracks task progress, requests new containers when tasks fail, and marks the job complete when every task has succeeded. Speculative execution launches duplicate copies of slow-running tasks to guard against stragglers.