Why it matters

GLU FFNs are standard in modern LLMs. Understanding shapes design.

Advertisement

The architecture

FFN(x) = down(gate(up(x)) * activation(up(x))). Two projections up + one down.

GLU FFN structureUp projectionW_up + W_gateActivationsigma or SiLUDown projectionW_downTwo up-projections + gate multiplication + one down-projection
GLU flow.
Advertisement

How it works end to end

Params: 3 matrices vs 2 for ReLU FFN.

Adjust hidden dim to keep param count.

Quality: consistently better.