Why it matters

3D parallelism is frontier scaling. Understanding shapes design.

Advertisement

The architecture

TP within node (NVLink).

PP across nodes.

DP wraps everything.

3D parallelismTP intra-node8-way NVLinkPP inter-nodestagesDP outerreplicateFrontier models: 8 TP + 8-16 PP + hundreds DP; 4D adds EP for MoE
3D parallelism.
Advertisement

How it works end to end

TP = 8 (NVLink domain).

PP = 8-16 typical.

DP = fills remaining.

MoE adds EP → 4D or more.