Why it matters

Tensor parallel is key for large layer models. Understanding shapes intra-node parallelism.

Advertisement

The architecture

Column parallel: split matmul weights by column. All-gather output.

Row parallel: split by row. All-reduce output.

Tensor parallel patternsColumn parallelsplit by output dimRow parallelsplit by input dimAll-reduce or gathercombine outputsSequential column + row within transformer block avoids intermediate all-gather
TP within transformer.
Advertisement

How it works end to end

Combined pattern in transformer: column parallel Q, K, V projections + row parallel output projection.

Communication: all-reduce after each block. NVLink bandwidth critical.

Within node only: cross-node too slow.