Why it matters

Gradient checkpointing enables larger models. Understanding shapes memory.

Advertisement

The architecture

Optimal checkpoint spacing: sqrt(L) for L layers.

Memory O(sqrt(L)) vs O(L).

Compute 1.33x.

Gradient checkpoint mathLayers Lin modelCheckpointsspaced sqrt(L)Memory + computeO(sqrt L) + 1.33xChen et al. 2016; optimal spacing; PyTorch supports; large model foundation
Gradient checkpoint math.
Advertisement

How it works end to end

Chen et al. 2016.

Optimal spacing = sqrt(L).

Selective for expensive activations.