Core concept
GPipe, PipeDream, 1F1B schedules. Trade micro-batch count vs bubble.
Advertisement
How it works
Complex implementation. Interleaved schedules further reduce bubble but complicate.
Advertisement
Trade-offs + gotchas
Across-node parallelism. Combines with tensor + data.