In data-parallel training, every step ends with an all-reduce of the gradients, and that collective is pure overhead — it produces no new arithmetic, it only makes the replicas agree. The obvious optimization is to run it while the GPU is still doing useful work, so the wall-clock cost disappears into the backward pass. Every serious training framework does this, and it is the single largest reason multi-GPU scaling looks as good as it does. But “overlap” is quietly one of the most over-claimed words in distributed training. A collective is not a background process running on someone else’s hardware; it is kernels on your SMs and traffic on your memory system, competing with the very compute it is hiding behind. This piece is about the overlap problem specifically: how bucketing creates the opportunity, what bounds it, what it costs, and how to tell in a profile whether you actually got it.
Why you fire the collective during backward, not after
The naive implementation runs the whole backward pass, then all-reduces one giant gradient buffer. It is correct and it is easy to reason about, and it leaves the entire communication time exposed: for the duration of that collective the GPU has no compute queued, so step time is literally backward time plus communication time. As you scale out, the second term grows while the first stays fixed, and scaling efficiency collapses.
The insight that fixes it is that gradients are not all produced at the same moment. Backpropagation walks the graph from the loss toward the inputs, so a layer’s gradient is final the instant its backward node completes — there is nothing left in the step that will change it. That gradient is therefore ready to be reduced immediately, while the rest of backward is still running. Instead of one collective at the end with nothing to hide behind, you issue many collectives spread across the backward pass, each with the remaining backward work available as cover.
Bucketing: the unit of overlap
Reducing each parameter tensor the moment it is ready would be the finest possible granularity and it performs badly. Real models have thousands of parameter tensors, many of them tiny — biases, layer-norm scales — and a collective on a few kilobytes is dominated by fixed costs: the launch, the handshake across ranks, the per-message protocol overhead. You would spend the step doing latency, not bandwidth.
Gradient bucketing is the compromise. The framework groups parameters into fixed-capacity buckets of contiguous memory; as each parameter’s gradient is produced, it is written into its slot and a counter for that bucket decrements. When a bucket’s last gradient lands, the bucket is complete and exactly one all-reduce is launched for the whole buffer. PyTorch DDP does this by default with a bucket_cap_mb that defaults to 25 MB. The bucket, not the tensor and not the model, is the atom of both communication and overlap.
Bucket size is a trade, and both directions hurt
Bucket capacity is the one knob that most directly shapes the overlap profile, and it pulls two ways. Larger buckets mean fewer collectives, each moving more bytes, which amortizes launch and protocol overhead and pushes the interconnect toward its asymptotic bandwidth — large messages are simply more efficient per byte. But a large bucket completes later, because it waits on more gradients, so the collective starts nearer the end of backward and has less compute left to hide behind.
Smaller buckets invert everything: reduction begins earlier and the communication is spread more evenly across the backward pass, which is exactly what you want for overlap, but you pay more fixed cost per byte and you issue many more collectives. Push it far enough and the aggregate communication time grows faster than the overlap window you bought. The default is a reasonable middle; the point is that tuning it is a genuine optimization problem, not a bigger-is-better dial.
The dependency order that bounds how early you can start
Because backward runs from the loss inward, gradients become ready in roughly reverse layer order: the last layer’s gradients appear first, the embedding’s appear last. Frameworks exploit this by assigning parameters to buckets in reverse order of the forward parameter list, so that bucket boundaries line up with the order gradients actually arrive. DDP goes further and rebuilds that bucket order after the first iteration, once it has observed the real completion sequence rather than guessing from declaration order.
This ordering is also the hard bound on overlap. The bucket containing the earliest layers can only be complete when backward is essentially finished — by construction there is almost no compute remaining to hide it behind. No amount of tuning removes that; it is structural. The best achievable outcome is that every bucket except roughly the last one is fully covered, and the final bucket’s collective is exposed. That residue sets the floor on your step time.
A separate communication stream is necessary and not sufficient
For a collective to run concurrently with compute at all, it must be issued on a different CUDA stream than the compute kernels — work on one stream is ordered, and a collective queued behind the backward kernels would simply wait its turn. Frameworks therefore keep a dedicated communication stream, with events recorded so the collective waits on the gradient-producing kernel and the optimizer waits on the collective. That mechanism is table stakes.
What people expect from it, and do not get, is scheduling guarantees. Stream priority is not preemption. A high-priority stream influences which thread blocks the GPU schedules next, when resources free up; it cannot evict blocks that are already resident on the SMs. If a large GEMM has filled the device, a newly launched high-priority collective waits for blocks to retire. Priority shortens the queueing delay before the collective gets going — it does not let it jump an occupied machine.
The contention nobody budgets for
Here is the part that surprises people the first time they measure it. A collective is not free-floating DMA handled by some off-chip engine. NCCL implements its collectives as kernels: they occupy SMs, consume registers and shared memory, and are scheduled against your compute kernels for the same resources. Running a collective concurrently with a GEMM therefore removes SMs from that GEMM, and the GEMM gets slower.
Even when the collective is configured to use few SMs, a second contention channel remains: it reads and writes gradient buffers in HBM and moves data through L2, so it consumes memory bandwidth that a bandwidth-bound kernel also wants. This is why overlap can appear perfect in a timeline and still return less than you expected. The number of channels the collective uses is the mechanism to trade here — fewer channels means fewer SMs occupied and less interference, at the cost of lower collective bandwidth.
Real overlap versus apparent overlap in a profile
Two bars sitting side by side on a profiler timeline prove concurrency, not benefit. The classic false positive is a trace showing beautiful overlap while step time refuses to move, because the SM and bandwidth the collective stole cost exactly what the overlap saved. If you only ever look at whether the bars line up, you will optimize this for a long time and gain nothing.
Two measurements settle it. The first is kernel dilation: the duration of a given compute kernel when a collective is running concurrently, compared with its duration in a run where communication is disabled or serialized. That difference is the real price of overlap. The second is exposed communication time — the wall-clock during which a collective is in flight and no compute kernel is executing. That is the quantity overlap exists to shrink, and it is the one to track step over step.
When the collective cannot be hidden at all
Overlap has a ceiling set by arithmetic, and it is worth knowing which side of it you are on. If the total communication volume per step, divided by the achievable interconnect bandwidth, exceeds the backward pass’s compute time, then no scheduling can hide it: there is not enough compute in the step to cover the bytes. Small models on many ranks, or fast accelerators behind a slow fabric, land here routinely, and the fix is not tuning but reducing the volume or raising the batch size.
The other failure is local rather than global. Even when the totals work out, the last bucket’s collective has nothing behind it, and a step whose backward pass ends abruptly leaves that tail fully exposed. Gradient accumulation with communication skipped on non-boundary microbatches is another instance: you save many collectives, then concentrate one very large, entirely exposed collective at the boundary step.
What perturbs the overlap window in practice
Several ordinary settings quietly change how much of the backward pass is available as cover. Asking the framework to search for unused parameters forces a graph traversal at the start of backward to decide which gradients will ever arrive, which delays bucket completion and costs real time every step — leave it off unless the model genuinely has conditional branches. Declaring the graph static lets the framework trust its learned bucket order instead of re-deriving it. Having gradients view the bucket buffer directly avoids a copy into the bucket, removing work from the critical path between gradient production and launch.
Sharding the optimizer state and parameters changes the shape rather than the principle: the single all-reduce becomes a reduce-scatter during backward, and once parameters are sharded too, an all-gather of parameters around forward. There are then two collectives to hide, and the forward pass becomes a place overlap has to happen as well.