Why it matters

Hyena hierarchy is subquadratic attention alternative. Research direction.

Advertisement

The architecture

Long conv + data gates.

Recursive hierarchy.

Subquadratic.

Hyena hierarchy mathLong convlearnable filterData gatesinput-dependentRecursivehierarchicalPoli et al. 2023; research direction; competitive at moderate scale
Hyena hierarchy math.
Advertisement

How it works end to end

Poli et al. 2023.

Long convolutions.

Data-controlled gates.