Forward and Backward Arithmetic Counts

Forward and Backward Arithmetic Counts: experiment guide

This calculator counts arithmetic for one bias-free dense layer, Y = XW. It deliberately exposes tensor dimensions rather than treating every model parameter as a guarantee of identical computation. Batch size multiplied by Sequence length gives the number of token rows T in X. Input width I and Output width O determine X shape [T, I], W shape [I, O], and Y shape [T, O]. There are I times O stored weight parameters. All four dimension controls accept positive integers; zero, negative, fractional, empty, or nonfinite dimensions produce an error instead of a misleading cost. Reset parameters restores batch 1, sequence 4, input width 3, and output width 2.

Worked details 1

The forward calculation computes T times O dot products, each of length I. A straightforward dot product uses I multiplications and I minus one additions when the first product initializes the sum. Consequently forward multiplication count is TIO and addition count is TO(I − 1). The weight gradient dW = X transpose times dY has the same multiplication count, but its sums run over T rows, giving IO(T − 1) additions. If Compute input gradient is checked, dX = dY times W transpose also uses TIO multiplications and TI(O − 1) additions. The table exposes all three terms separately, with the input-gradient row becoming zero when that computation is disabled.

Worked details 2

At the reset dimensions, T = 4, I = 3, and O = 2. Forward requires 24 multiplications and 16 additions, totaling 40 scalar operations. The weight gradient uses 24 multiplications and 18 additions, totaling 42. The input gradient uses 24 multiplications and 12 additions, totaling 36. Backward therefore totals 78 and the whole forward-plus-backward calculation totals 118. Disable Compute input gradient to obtain a total of 82, as might apply when no derivative into an earlier layer is required. Recompute forward in backward adds one complete extra forward evaluation: with the default input-gradient checkbox enabled, total arithmetic becomes 158. This checkbox models one explicit recomputation, not every checkpointing schedule.

Worked details 3

The separate Two-per-MAC convention reports two operations for every multiply-accumulate slot, including a possible initial addition to zero. At the reset settings it reports 144 FLOPs rather than the exact scalar expression count of 118. Both figures have explicit counting rules; neither is a processor instruction count. For a sharp boundary test, set all four dimensions to one. Forward, weight gradient, and input gradient each need one multiplication and no additions, giving a total of three, whereas the two-per-MAC convention gives six. The familiar approximation that backward costs twice forward emerges for large dense dimensions when both gradients are computed; this small example explains why it is not an exact identity for every shape.

Worked details 4

The bars compare the calculated counts and are immediately updated when a control changes. They do not represent elapsed time and require no animation timer. The text and table provide every result without relying on bar length or color. The scope excludes attention score matrices, nonlinear activations, loss evaluation, biases, optimizer updates, embedding lookup, data movement, kernel launch overhead, sparsity, and communication. Sequence length here counts independent dense-layer rows; it does not introduce attention’s quadratic sequence cost. No CPU throughput constant is assumed, so the page makes no fabricated seconds-per-step prediction or claim about actual training performance.