Why it matters

SP enables long context. Understanding shapes long-context training.

Advertisement

The architecture

Split sequence.

Layernorm + activation sharded.

All-gather / reduce-scatter as needed.

Sequence parallelism flowSplit sequenceacross GPUsShard LN + actsreduce activation memRing for attentionlong contextMegatron SP shards LN + dropout; ring attention shards attention itself
SP.
Advertisement

How it works end to end

Megatron SP: after TP, further shard activations.

Ring attention: full attention sharding.

Combine with TP.