Why architecture matters here

Absolute pos limits long-seq extrapolation. Architecture matters because bias + buckets + extrapolation compose.

Advertisement

The architecture: every piece explained

The top strip is basics. Attention Q K. Distance. Bias table. Add to logits.

The middle row is variants. T5 buckets. Shaw edges. ALiBi. Extrapolation.

The lower rows are ops. Cost. Metrics. Ops — model class + inference length + cache.

Relative position embeddings — bias + T5 style + Shawencode distance instead of absolute posAttention Q Kno abs posDistance i-jsignedBias tablelearnableAdd to logitsbefore softmaxT5 bucketslog-scaleShaw edgesper relationALiBilinear penaltyExtrapolationbeyond train lenCostextra table + addsMetricslong-seq qualityOps — model class + inference length + cachebucketshawalibiextendbudgetwatchwatchoperateoperate
Relative position: distance-based bias into attention.
Advertisement

End-to-end flow

End-to-end: compute Q K^T. Compute distance matrix. Lookup bias in table (or apply ALiBi slope). Add to logits. Softmax. Attention proceeds.