Selective Activation Recomputation

Why it matters

Selective recompute optimizes trade-off. Understanding shapes advanced training.

The architecture

Store: MLP + norm activations (small).

Recompute: attention softmax + intermediate (large).

How it works end to end

Megatron-LM introduced.

Attention softmax + P dropout stored else large.

~5% compute vs 30% for full checkpoint.