Why it matters

Tensor parallel shapes LLM training. Understanding shapes design.

Advertisement

The architecture

Column-parallel: weights split by out dim.

Row-parallel: weights split by in dim.

All-reduce between column + row.

Tensor parallel structureColumn-parallelsplit output dimRow-parallelsplit input dimAll-reducecombine partial productsShoeybi et al. 2019 Megatron; standard for LLM training on NVLink domain
Tensor parallel math.
Advertisement

How it works end to end

Shoeybi et al. 2019 Megatron-LM.

NVLink domain (8 GPUs typical).

All-reduce after each block.