Inference Latency: Conditional Resource Bounds

Inference Latency: Conditional Resource Bounds: experiment guide

This direct calculator describes one hypothetical dense decode step. Every sequence in the batch receives one new token. Dense parameters supplies the number of weight parameters in millions, and Packed weight bits supplies the assumed number of bits stored per parameter. Decode batch size supplies the number of sequences sharing that weight read. The model assumes all weights are read once from the selected memory level per batch step. It assumes perfect reuse across the batch and no reuse between steps. These are declared traffic assumptions, not universal properties of inference. Packed storage excludes quantization scales, zero points, alignment and metadata.

Worked details 1

Other traffic per sequence supplies a user estimate in decimal megabytes per step. It can include K/V reads and writes, activations and any additional data movement you choose to account for. It is not a cache-capacity setting and is not derived from sequence length. Assumed bandwidth is a ceiling in decimal gigabytes per second. Assumed compute ceiling is a compatible arithmetic throughput in billions of operations per second, counting a multiply and add as two operations. The default numerical values are hypothetical and are not specifications for a named CPU. Explicit serial overhead adds a declared nonoverlapping time after the resource floor; its initial value is zero. There is no hidden thirty-percent multiplier or claim that a fixed overhead is realistic.

Worked details 2

The calculation uses weight megabytes = million parameters times bits divided by eight. Total traffic adds batch size times other traffic. Dense operations are approximately two times parameters times batch, omitting attention, normalization, activation functions, dequantization and sampling. Memory milliseconds equal total megabytes divided by gigabytes per second: the decimal unit conversion supplies exactly the required thousandfold seconds-to-milliseconds factor. Compute milliseconds equal total GFLOPs divided by the GFLOP/s ceiling, times one thousand. With ideal overlap the kernel floor is the larger resource time, not their sum. The step floor adds the explicit serial overhead. Aggregate token throughput cannot exceed batch times one thousand divided by this conditional step floor; each individual sequence receives only one token per step.

Worked details 3

At the reset settings, one thousand million parameters stored at four bits occupy 500 MB. A batch of one with 32 MB of other traffic moves 532 MB. At 50 GB/s the memory floor is 10.64 ms. The dense calculation requires 2 GFLOPs, giving a 2 ms compute floor at 1000 GFLOP/s. The step floor is therefore 10.64 ms, and the conditional aggregate ceiling is approximately 93.984962 tokens/s. Double bandwidth to 100 GB/s: the floor becomes 5.32 ms, while the compute floor remains 2 ms. Set bandwidth to 1000 GB/s and compute becomes the limiting resource at 2 ms; increasing bandwidth further cannot reduce that floor.

Worked details 4

Reset, then change the batch to eight. Weight traffic remains 500 MB, other traffic becomes 256 MB, and dense work becomes 16 GFLOPs. Memory takes at least 15.12 ms and compute at least 16 ms, so compute now limits the conditional step floor. Aggregate ceiling is 500 tokens/s, while each sequence is limited to 62.5 tokens/s. Add 4 ms of explicit overhead to make the step floor 20 ms. Reset parameters restores all initial values. Zero or negative throughput is rejected rather than yielding a misleading finite prediction; fractional bit widths or batch counts are also rejected. The chart and table update immediately, so no teaching timer is needed. These bounds require the stated traffic, work and compatible resource ceilings. Cache residency, sparse routing, extra operations, contention and incomplete overlap can invalidate or loosen the assumptions; actual performance requires measurement.