Why architecture matters here
Activation choice matters because it appears billions of times per training / inference step. Small approximations compound; missing fusion costs bandwidth.
The architecture: every piece explained
The top strip is the family. Preactivation x is linear output. ReLU family is fast, non-smooth. GELU / SiLU are smooth gates common in transformers. Gated (GeGLU/SwiGLU) add elementwise gate for capacity.
The middle row is implementation. Approximation: GELU with erf is exact; tanh approx faster. Numerics: fp32 boundary at extremes. Kernel fusion: combine with linear + norm to save bandwidth. Backprop: elementwise gradient.
The lower rows are ops. Model choice per layer. Metrics throughput + quality. Ops covers dtype + hardware fit.
End-to-end flow
End-to-end: a modern LLM uses SwiGLU in the FFN. Kernel fuses linear + gate + activation. Throughput 12% higher than unfused. Quality on eval beats plain GELU by 1-2 points per benchmark. bf16 in, fp32 accumulator internally.