Why it matters

Effective batch shapes gradient quality. Understanding shapes design.

Advertisement

The architecture

Micro-batch: per-GPU per-forward.

World: GPU count.

Grad accum: passes before update.

Product = effective.

Effective batch equationMicro-batchper-GPUWorld sizeGPUsGrad accumsteps before updateAll three levers combine multiplicatively; adjust as memory allows
Effective batch.
Advertisement

How it works end to end

Choose target effective batch.

Fit micro-batch to memory.

Use world size for parallel.

Grad accum fills gap.