Why it matters

Linear attention scales to long sequences. Quality trade-off remains.

Advertisement

The architecture

Kernel feature map φ.

Compute φ(K)^T V once.

Linear in N.

Linear attention mathKernel φfeature mapφ(K)^T Vcompute onceLinearin sequence NKatharopoulos 2020; ELU, ReLU, exp kernels tried; quality gap remains
Linear attention math.
Advertisement

How it works end to end

Katharopoulos 2020.

ELU, ReLU kernels.

Performer's random features.