Why it matters
FlashAttention is the standard implementation for modern LLM inference and training. Understanding it explains why long-context is now feasible and why hardware matters.
The architecture
Key insight: softmax can be computed online. Split Q, K, V into tiles. For each tile, compute partial attention, update running max and normalization. Never store full attention matrix.
Everything happens in SRAM (fast). HBM only accessed for input Q, K, V and output.
How it works end to end
Backward pass: FlashAttention has a specific backward-compatible algorithm. Both forward and backward avoid materializing attention matrix.
FlashAttention-2 improves throughput further with better GPU utilization. FlashAttention-3 targets Hopper-specific optimizations.
Enables long-context: 128K, 1M context now feasible thanks to FlashAttention memory efficiency.