Why architecture matters here
Position choices matter because they decide long-context extrapolation, kernel fusion opportunities, and inference throughput.
The architecture: every piece explained
The top strip is types. Token embeddings alone are order-agnostic. Sinusoidal uses fixed sin/cos. Learned is a trainable table. Relative encodes offset directly in attention.
The middle row is modern. RoPE rotates Q + K based on position. ALiBi adds linear bias to attention scores. Long-context extension — PI, NTK, YaRN. Kernel fusion in attention.
The lower rows are practice. Model choice per architecture. Metrics long-context recall. Ops — training + serving alignment.
End-to-end flow
End-to-end: Llama-family uses RoPE with base 10000. Attention kernel fused with rotation. Extension via YaRN for 128k context. Long-doc QA eval shows preserved recall.