Core concept
Simple to think about. Communication between layers. Bubble time (idle GPUs during forward-only or backward-only phases).
Advertisement
How it works
Naive model parallel: only 1 GPU active at a time. Poor efficiency. Pipeline parallel fixes.
Advertisement
Trade-offs + gotchas
Very large models that don't fit on one GPU. Foundational for LLM training.