Core concept
Introduced by Megatron-LM. Splits Q/K/V by heads. FFN by hidden dim.
Advertisement
How it works
Communication per forward + backward. Requires high-bandwidth interconnect (NVLink typical).
Advertisement
Trade-offs + gotchas
Within-node parallelism. Combined with data + pipeline for multi-node.