Why it matters

MapReduce shaped big-data era. Understanding shapes distributed patterns.

Advertisement

The architecture

Split input into chunks.

Map function per chunk (parallel).

Shuffle by key.

Reduce per key.

MapReduce data flowSplit + mapchunks + parallelShuffleby keyReduceaggregate per keyDean + Ghemawat 2004; Hadoop MapReduce dominant then; Spark evolved model
MapReduce.
Advertisement

How it works end to end

Dean + Ghemawat 2004.

Hadoop MapReduce first, Spark evolved.

Word count canonical example.

Restricted programming model.