Why architecture matters here

Placement failures show up as unstable training, gradient blowup, or slow convergence. Architecture matters because init + placement + warmup are one decision.

Advertisement

The architecture: every piece explained

The top strip is definitions. Post-norm x + norm(sublayer(x)). Pre-norm x + sublayer(norm(x)). Gradient flow — pre-norm has a clean residual path. Stability — large models pre-norm-friendly.

The middle row is training details. LR warmup need post-norm requires more. DeepNet init scaling for post-norm. Sandwich norm hybrid. Final LN still at head.

The lower rows are practice. Model choice modern LLMs pre-norm. Metrics loss smoothness. Ops — init + warmup + tests.

Pre-norm vs post-norm — stability + gradient flow + init + DeepNetthe placement decision that decides trainabilityPost-normx + norm(sublayer(x))Pre-normx + sublayer(norm(x))Gradient flowresidual path cleanStabilitylarge models trainLR warmup needpost-norm requiresDeepNetinit scaling for post-normSandwich normhybrid variantFinal LNstill at headModel choicemodern LLMs pre-normMetricsloss curve smoothnessOps — init + warmup + testswarmupscalehybridaddchoosemeasuremeasureoperateoperate
Pre-norm vs post-norm placement of LayerNorm.
Advertisement

End-to-end flow

End-to-end: modern LLM uses pre-norm. Loss curve smooth from step 1. Warmup mild. Training scales to trillions without explosion. Compare to post-norm at same scale without DeepNet — early exploding gradients.