Why it matters

Selective recompute improves trade-off. Understanding shapes advanced training.

Advertisement

The architecture

Identify large activations.

Store cheap ones.

Recompute expensive.

~5% compute + significant memory.

Selective recompute strategyIdentify largeexpensive to storeStore cheap onesMLP + normRecompute expensiveattention scores + dropoutKorthikanti et al. 2022 Megatron; ~5% compute for major memory savings
Selective recompute math.
Advertisement

How it works end to end

Korthikanti et al. 2022.

Attention scores + dropout expensive to store.

MLP + norm cheap.

~5% compute overhead.