Why it matters

Parallel matmul is HPC + LLM foundation. Understanding shapes distributed training.

Advertisement

The architecture

Partition matrices across procs.

Rotate blocks.

Multiply + accumulate.

2.5D adds replication.

Parallel matmul algorithmsPartitionblock matricesRotate + multiplySUMMA / Cannon2.5D adds replicationtrade memory for commCannon 1969, SUMMA 1995, 2.5D 2011; TP + FSDP in LLM training use variants
Parallel matmul.
Advertisement

How it works end to end

Cannon: torus rotation.

SUMMA: broadcast panels.

2.5D: replicated with less comm.

Tensor parallelism in LLMs.