Training memory allocation ledger

Declare the arrays and allowances

One unsharded replica; all parameters trainable. Enter actual parameter units and integer MiB allowances. No architecture or activation size is inferred.

Blue bars are components; the rose bar is their sum. This is a direct numerical tool without an animation or benchmark timer.

Reference: PyTorch memory allocation and optimizer lifetimes.

Account for arrays, not a guessed architecture

This calculator adds explicitly declared memory allocations for one unsharded training replica. Parameters is the number of scalar trainable parameters, not a count of layers or millions unless you enter that many units. The model does not infer hidden width or activation size from parameter count. Weights, gradients, an optional master copy and optimizer state are counted independently. Saved activations, temporary workspace and other allocations are supplied in MiB. The result is an allocation scenario, not a measurement of a running model or a promise that training will fit.

Choose the storage convention explicitly

Weight bytes and Gradient bytes each select two or four bytes per parameter. These are payload widths only: the calculator does not emulate BF16, FP16, FP32, numerical overflow or training accuracy. Separate FP32 master copy adds another four bytes per parameter when checked. With four-byte weights, checking it deliberately counts a second full-precision array; uncheck it if your implementation updates the existing weights directly. Optimizer allocation offers no persistent state, one FP32 momentum buffer, or two FP32 Adam moment buffers. They add zero, four or eight bytes per parameter respectively. These choices describe arrays, not every implementation of the named optimizer. State datatypes, lazy allocation and fused update behavior vary. All parameters are assumed trainable and receive gradients and the selected state.

Supply activations and workspace rather than hiding a formula

Saved activations accepts a nonnegative integer number of MiB for tensors retained for backward. Temporary workspace is another nonnegative MiB allowance for intermediate operations. Other allocations / reserve covers additional buffers or deliberate headroom. Enter a measured or separately justified scenario for these values. Activation checkpointing may change activation lifetimes and recomputation, but this tool offers no invented universal saving factor; enter the revised allocation yourself. These three allowances are summed as simultaneously present. If their measured peaks occur at different times, summing them can overestimate the actual simultaneous peak. Conversely, omitted buffers or allocator fragmentation can make the estimate too small.

Reproduce the byte-exact boundary

Enter 1048576 parameters, two-byte weights, two-byte gradients, a separate FP32 master copy, and two FP32 Adam moments. Set all three MiB allowances to zero. The parameter arrays consume sixteen bytes per parameter, totaling 16777216 bytes or exactly sixteen MiB. Set Capacity to sixteen MiB: the status is exact equality and headroom is zero. Set it to fifteen: the ledger exceeds capacity by one MiB. Set it to seventeen: declared headroom is one MiB. Uncheck the master copy and the total falls to twelve MiB. Select no optimizer state and it falls to four MiB with the master still disabled. Every change must agree with the component table.

Use every control and inspect invalid inputs

Parameters accepts integers from one through one trillion. Capacity accepts one through 1048576 MiB, and the three allowances accept zero through that same maximum. Blank, fractional, nonfinite and out-of-range numeric values are rejected; the chart and table clear instead of retaining an earlier calculation. Reset defaults restores one million parameters, two-byte weights and gradients, a separate master copy, Adam moments, sixty-four MiB activations, thirty-two MiB workspace, 128 MiB reserve and 1024 MiB capacity. Full precision calculation reveals the configuration, each component and signed headroom in bytes. The table remains the numerical text alternative to the chart. On a phone, focus the table and scroll horizontally to inspect the byte column.

Interpret units and limits carefully

One MiB means 1048576 bytes, not one million bytes. All allowed byte totals remain exact safe integers in JavaScript; displayed MiB values are rounded for readability. The chart uses one common scale for component bars and the total, rather than separately normalizing each component. Capacity is your declared budget, not memory detected from this browser. A within-budget result only compares these allocations against that number. CUDA contexts, communication buffers, distributed sharding, offloading, sparse gradients, quantized optimizer metadata, optimizer temporaries and allocation reuse are not modeled unless represented in an allowance. No GPU is accessed, and there is no simulated clock or fabricated benchmark. Use a profiler and the actual implementation to validate a deployment decision.