Why it matters

TP enables larger layers than fit on one GPU. Understanding shapes design.

Advertisement

The architecture

Column-parallel MLP.

Row-parallel next layer.

All-reduce within layer.

Tensor parallelism flowColumn splitMLP weightsRow splitnext weightsAll-reduceactivationMegatron paper; NVLink required for reasonable perf; degree matches NVLink domain
TP.
Advertisement

How it works end to end

Column-parallel + row-parallel pairs.

Cross-GPU comm each layer.

Best within NVLink domain (8 GPUs).