Pipeline Parallelism — Layer-Wise Model Splitting

Why it matters

Pipeline parallel is key for very large models. Understanding shapes multi-node training.

The architecture

Split model into stages. Each stage on different GPU.

Micro-batch: divide batch into smaller pieces. Pipeline them.

How it works end to end

Bubbles: idle time during pipeline fill/drain. Larger with fewer micro-batches.

Schedules: GPipe (simple), 1F1B (interleaved forward+backward), interleaved 1F1B (smaller bubble).

Combine with data + tensor parallel for 3D parallelism.