Why it matters

Sequence parallel shapes long context. Understanding shapes training.

Advertisement

The architecture

Shard activations along sequence.

Reduce-scatter + all-gather for cross-shard.

Combines with TP.

Sequence parallel structureShard along seqon TP-partitioned activationsReduce-scatter + all-gathercross-shard opsCombine with TPfurther memory saveKorthikanti et al. 2022 Megatron SP; shards LayerNorm + dropout after TP
Sequence parallel math.
Advertisement

How it works end to end

Korthikanti et al. 2022 Megatron SP.

LayerNorm + Dropout activation sharded.

Requires TP.