Why it matters

Flash Attention shapes long context. Understanding shapes optimization.

Advertisement

The architecture

Tile Q + K + V.

Online softmax.

Accumulate output tile.

SRAM-resident computation.

Flash Attention mathTile Q + K + Vload to SRAMOnline softmaxblock reductionAccumulate O tileno HBM materializationDao et al. 2022; FA-2 improved; FA-3 for H100 async/FP8
Flash Attention math.
Advertisement

How it works end to end

Dao 2022 + FA-2 + FA-3.

O(N) memory instead of O(N^2).

Higher arithmetic intensity.