Why it matters

Linear attention is theoretical family; practical variants growing.

Advertisement

The architecture

Kernel: phi(Q) phi(K)^T.

Recurrent form: state matrix.

Linear attentionFeature mapphiComputephi(Q) phi(K)^TRecurrent formstate matrixPerformer, Linformer, Reformer, gated linear (Mamba-2) are variants
Linear attention.
Advertisement

How it works end to end

Performer: random features.

Linformer: low-rank projection.

Gated linear attention: input-dependent decay (Mamba-2).