Why it matters

Memory math shapes training feasibility. Understanding shapes design.

Advertisement

The architecture

Params: 2 bytes (bf16) or 4 (fp32).

Grads: same as params.

Optim (AdamW): 8 bytes/param.

Activations: batch * seq * dim.

LLM memory breakdownParams + grads4-6 bytes/paramOptim state8+ bytes/paramActivationsbatch * seq * hidden70B model: 140GB just params bf16; needs sharding at that scale
Memory math.
Advertisement

How it works end to end

Total: ~16 bytes/param fp32 mixed prec + activations.

Example: 70B * 16 bytes = 1.1TB + activations.

Must shard across GPUs.