Why it matters

Activation checkpointing enables larger models. Understanding shapes memory trade-offs.

Advertisement

The architecture

Store activations only at checkpoints.

Between checkpoints: recompute forward in backward.

Activation checkpointingForwardcheckpoints onlyBackwardrecompute segmentsResultcut memory 5-10xPyTorch torch.utils.checkpoint; FSDP + activation ckpt commonly combined
Activation ckpt.
Advertisement

How it works end to end

PyTorch: torch.utils.checkpoint.

Segments: layer or block granularity.

Trade: 30% compute for 5-10x memory.