Why architecture matters here
Sparse attention fails when the mask doesn't capture the information the task needs. A model trained with dense attention deployed with sparse regresses. Kernels without sparse support negate the throughput gain.
The architecture matters because choice of pattern + kernel decides speed and quality.
The architecture: every piece explained
The top strip is the primitives. Full attention is the O(N²) baseline. Local window attends within a sliding window. Strided attends every k tokens. Global tokens are special positions that attend everywhere.
The middle row is the compositions. Block sparse works with block-level masks. Longformer combines local + global. BigBird adds random attention. Kernel support — block-sparse GEMM — makes the theoretical savings real.
The lower rows are practice. Training vs serving must use same masks. Metrics track recall + throughput. Ops covers per-model pattern, eval, and kernel matrix.
End-to-end flow
End-to-end: a long-document model uses Longformer pattern with window=512 + global tokens on sentence boundaries. Training + serving use block-sparse kernels. On a 16k context, throughput 3x vs dense, quality on long-doc QA within 1 point. Ops verifies pattern in both stages.