Why it matters

ALiBi enables length extrapolation. Understanding shapes long-context.

Advertisement

The architecture

Attention score: Q K^T / sqrt(d) - m_head * (i - j).

m_head: per-head slope.

ALiBi bias structureAttention scoreQ K^TAdd bias-m * distanceSoftmaxattention weightsPer-head slopes m_h = 2^(-8h/H) form geometric progression
ALiBi math.
Advertisement

How it works end to end

Slopes: 2^(-8h/H) for head h of H heads.

No position embedding needed.

Extrapolates beyond training length.