Why it matters

RetNet aims for best of both worlds. Research direction.

Advertisement

The architecture

Retention mechanism.

Parallel or recurrent.

Chunkwise hybrid.

RetNet mathRetentionparallel formRecurrent formfor inferenceChunkwisehybrid modeSun et al. 2023 Microsoft; ecosystem building; research direction
RetNet math.
Advertisement

How it works end to end

Sun et al. 2023 Microsoft.

Three modes.

Research direction.