Why it matters

Pipeline parallel shapes LLM training. Understanding shapes design.

Advertisement

The architecture

Layers split into stages.

Micro-batches flow through stages.

1F1B / interleaved reduces bubble.

Pipeline parallel schedulesStageslayers per stageMicro-batchesflow through1F1B / interleavedreduce bubblesHuang et al. GPipe + Narayanan PipeDream + Megatron interleaved 1F1B
Pipeline parallel math.
Advertisement

How it works end to end

GPipe + PipeDream + interleaved 1F1B.

Micro-batch size trade-off.

Cross-node comm minimal.