Why it matters

Decode math shapes throughput. Understanding shapes ITL.

Advertisement

The architecture

Per token: read all N params + KV.

Time = params_bytes / HBM_BW.

Batch amortizes reads.

Decode mathPer-token readN * bytes weightsHBM bandwidthTB/sTokens/secamortized in batchH100: 3.35 TB/s HBM; 70B bf16 = 140GB; ~24 tok/s per stream single-batch
Decode math.
Advertisement

How it works end to end

Bytes per token: N * bytes/param.

Batching amortizes: multiple streams share weight reads.

Speculative decoding improves.