Why it matters

Micro-batch shapes memory. Understanding shapes design.

Advertisement

The architecture

Memory: proportional to micro-batch.

Throughput: peaks then plateaus.

Choose largest fitting memory.

Micro-batch selectionMemory profilevs micro-batchThroughput curveincrease then flatChooselargest fittingIn pipeline parallelism, small micro-batches reduce bubble but increase count
Micro-batch.
Advertisement

How it works end to end

Small: memory OK, bad throughput.

Large: OOM.

Sweet spot: largest fitting with buffer.

In PP: smaller better for bubbles.