Why it matters

FA-2 is standard attention. Understanding shapes performance.

Advertisement

The architecture

Tiling: work on blocks in SRAM.

Reduce non-matmul FLOPs.

Better sequence-parallel.

FlashAttention-2 improvementsTile blocksin SRAMFewer non-matmulkernel optsSequence parallelbetter GPU utilFA-3 further improves on H100 with async + FP8
FA-2 vs FA-1.
Advertisement

How it works end to end

Reduce operations that don't use tensor cores.

Better warp partitioning.

Sequence-parallel outer loop.

Result: 2x speedup, 70% peak FLOPs on A100.