Why it matters

Grad accum enables larger effective batch without more memory. Standard technique.

Advertisement

The architecture

Micro-batch forward + backward.

Do not zero grads.

After K: step + zero.

Grad accum flowMicro-batchforward + backAccumulateK iterationsStep + zeroone updateLoss should be divided by K for correct gradient scaling
Grad accum.
Advertisement

How it works end to end

K micro-batches per step.

Divide loss by K for scaling.

DDP: sync only on step.

Effective batch K * micro-batch.