Why it matters

Recomputation shapes memory + speed. Understanding shapes training.

Advertisement

The architecture

Forward: store checkpoints not all activations.

Backward: recompute segments.

Trade compute + memory.

Activation recompute mathStore checkpointsat segment boundariesRecompute segmentsin backwardTrade30% compute vs 5-10x memoryPyTorch checkpoint_wrapper; standard for large model training
Activation recomputation math.
Advertisement

How it works end to end

PyTorch torch.utils.checkpoint.

~30% compute overhead.

5-10x memory reduction.

Segment granularity choice.