Why it matters

Hyena explores non-attention direction. Understanding shapes alternatives.

Advertisement

The architecture

Implicit conv filter: parameterized MLP.

FFT for fast conv.

Gating: input-dependent.

Hyena structureImplicit filterMLP parameterizedFFT convO(N log N)Gatinginput-dependentLong convolutions + gating; O(N log N) via FFT
Hyena.
Advertisement

How it works end to end

Filter length equals sequence length.

FFT for O(N log N) conv.

Gating for selectivity.