Why it matters

Sliding window enables long context. Used in Mistral, Longformer.

Advertisement

The architecture

Mask: allow attention to last W tokens.

KV cache: keep only last W tokens.

Sliding windowSliding windowsize WMasklocal attention onlyKV cachesize W not NLayered: distant info flows via stacking many local-attention layers
Sliding window.
Advertisement

How it works end to end

Window size: 4k typical.

Distant info: flows through layer stack.

KV cache scales with W not N.