Why it matters

Activation choice affects quality. Understanding shapes design.

Advertisement

The architecture

ReLU: max(0, x). Simple, some dead neurons.

GeLU: x * Phi(x). Smooth, used in BERT, GPT-2.

SiLU: x * sigmoid(x). Smooth, used in Llama.

SwiGLU: gated SiLU. Current default.

Activation evolutionReLUmax(0,x)GeLUx * Phi(x)SiLU / SwiGLUx * sigmoid(x)Trend: smoother + gated; more params but better quality per param
Activation comparison.
Advertisement

How it works end to end

Trend: smoother activations + gating.

Small quality gains stack.

Compute overhead minimal.