Why it matters

Activation memory scales with batch + seq. Understanding shapes training limits.

Advertisement

The architecture

Per layer: several tensors.

Attention: batch * heads * seq^2.

FFN: batch * seq * 4*dim.

Activation memory piecesAttention scoresbatch * heads * seq^2FFN intermediatebatch * seq * 4dNorm + residualsbatch * seq * dimSequence-length quadratic in attention; FlashAttention removes storage of scores
Activation memory.
Advertisement

How it works end to end

Attention: O(batch * heads * seq^2) without FlashAttention.

With FA: no explicit storage.

FFN: 4x hidden intermediate.