Core concept
NCCL implements ring + tree algorithms. Ring: bandwidth-optimal for many workers. Tree: latency-optimal for few.
Advertisement
How it works
Communication overhead dominant in distributed training if network slow.
Advertisement
Trade-offs + gotchas
Every gradient sync in data parallel. Every batch.