Gradient Accumulation and Unequal Microbatches

Gradient Accumulation and Unequal Microbatches: experiment guide

This experiment trains a one-parameter linear predictor on six fixed samples. Inputs x are 1, 2, 3, 4, 5, and 6; each target is 2x. The prediction is weight times x, and each sample loss is half the squared prediction error. Its gradient with respect to weight is (weight − 2) times x squared. Initial weight changes all six gradients, and Learning rate determines the final SGD update. Microbatch size partitions the six samples in their listed order. A size of four deliberately creates unequal groups of four and two, exposing a normalization error that equal-size examples can hide.

Worked details 1

Normalization selects the rule used to combine the per-microbatch mean gradients. Weight by sample count multiplies each mean by that batch’s sample count divided by six. Mean of batch means instead divides each mean by the number of microbatches. Those rules agree for equal batch sizes but generally differ for an unequal final batch. Sum of batch means omits averaging across batches altogether, illustrating another common error. The table shows every factor and contribution as it is revealed. The full-batch reference always averages all six individual sample gradients, independently of your chosen policy. A correct result therefore has a numerical reference rather than relying on a decorative progress bar.

Worked details 2

Play distributes the microbatch calculations and one final optimizer update across sixty teaching seconds. Pause freezes the trace; Restart clears its progress and pauses while preserving the inputs. Speed chooses 0.25×, 1×, or 4× presentation, and Seek moves directly to a paused inspection point. The first microbatch completes after 60 divided by the number of batches plus one seconds. All microbatches finish before the final update at exactly sixty seconds. Weight remains constant throughout gradient accumulation. The final stage applies SGD once and clears the live gradient buffer; the table retains the accumulated values as a historical trace. Reset parameters restores weight 1, learning rate 0.01, size 2, and sample-count normalization. Parameter changes restart the calculation without autoplay.

Worked details 3

At the reset settings, the individual gradients are −1, −4, −9, −16, −25, and −36. The three microbatch means are −2.5, −12.5, and −30.5. Each contributes one third, so the effective gradient is −91/6, approximately −15.166667. Seek to sixty seconds: one SGD update produces weight 1.151667. Now change the microbatch size to four and repeat. With sample weighting the same final gradient and weight appear. Select Mean of batch means: the two means are −7.5 and −30.5, their unweighted average is −19, and the final weight is 1.19. Select Sum of batch means to get gradient −38 and weight 1.38.

Worked details 4

For boundary experiments, use size one or six to compare one sample per batch against the full batch. Set Initial weight to 2 to obtain exactly zero gradients under every policy, or set Learning rate to zero to preserve the weight despite nonzero accumulated gradients. Fractional microbatch sizes and nonfinite inputs are rejected. This fixture has no batch normalization, dropout, gradient clipping, mixed precision, or optimizer state, so its equivalence statement applies to the fixed separable loss and one SGD update shown here. Accumulation reduces the activations required at once in many implementations, but this arithmetic lesson does not measure memory consumption or guarantee equivalence for layers that couple examples.