Backprop architecture — chain rule, accumulator, memory tradeoffs, fused kernels

Why architecture matters here

Backprop failures come from OOM + gradient explosion. Architecture matters because memory + precision + comms compose.

The architecture: every piece explained

The top strip is the passes. Forward pass. Loss. Backward pass. Gradient accumulation.

The middle row is optimization. Activation checkpointing. Fused backward kernels. Precision. Distributed all-reduce.

The lower rows are ops. Optimizer step. Metrics. Ops — debugging + numerics + throughput.

End-to-end flow

End-to-end: forward saves activations. Loss computed. Backward runs; checkpointed layers recompute activations as needed. Gradients accumulated N steps then all-reduced. Optimizer step in bf16 with fp32 master weights. Grad norm monitored for stability.