Why it matters

Seq length shapes training limits. Understanding shapes design.

Advertisement

The architecture

Attention: O(N^2) FLOPs + memory (without FA).

FFN: O(N * hidden).

KV cache scales with N.

Seq length scalingAttentionN^2 compute + memFFNN * hiddenKV cacheN * layers * dimLong context needs FlashAttention + ring attention + smart position encoding
Seq length.
Advertisement

How it works end to end

FlashAttention: no memory blow-up.

Ring attention: split across GPUs.

Compute: 6ND FLOPs for training.