Why it matters

SwiGLU is default modern LLM FFN. Understanding shapes model design.

Advertisement

The architecture

SiLU: x * sigmoid(x). Smooth, nonzero gradients.

Multiply by gate.

SwiGLU FFNUp + gatetwo projectionsSiLU on gatesmooth activationMultiply + downfinal projectionConsistently outperforms ReLU and GeLU FFN at same params
SwiGLU components.
Advertisement

How it works end to end

Introduced by Shazeer 2020.

Adopted by PaLM, Llama, Mistral, Qwen.

Small quality gain compounds.