Why architecture matters here
The economics of serving are the reason to care. A 7B-parameter model in bf16 is 14GB of weights; in BitNet b1.58 it is closer to 1.5GB. That difference is not merely a storage convenience — it decides whether the model fits in a phone's memory budget, whether it runs on a CPU without a discrete accelerator, and how many concurrent requests a given card can hold. Because single-stream decode is memory-bound, the wall-clock latency of generating a token is dominated by how long it takes to stream the weights past the compute units once per token. Cut the bytes by 10x and you cut that time by nearly 10x, before any compute savings are counted. Architecture here is destiny: the numeric format sets the performance ceiling.
The compute side compounds the win. A conventional matmul multiplies and accumulates; the multiplier is the expensive, power-hungry part of the datapath. Ternary weights turn every multiply into a select-and-add: if the weight is +1 add the activation, if -1 subtract it, if 0 do nothing. On silicon that means adders instead of multipliers — smaller area, lower energy per operation, and a natural fit for the integer-heavy datapaths of mobile NPUs. This is why BitNet is discussed as a co-design target for hardware rather than only a software trick: the format was chosen partly because it maps to cheap arithmetic.
But the trade-off is real and it is paid at training time, not inference time. Ternarizing a pretrained model post-hoc destroys it — three levels is far too coarse to approximate a distribution the network built around continuous weights. BitNet only works because the forward pass quantizes during training while gradients flow to a full-precision master copy through a straight-through estimator, so the optimizer learns weights whose ternary projection is what the loss actually sees. That means you cannot 'convert' an existing checkpoint; you commit to the scheme before the first training step. The bet is expensive to place and expensive to unwind.
Understanding the mechanism changes how a team evaluates it. The right question is never 'how much accuracy did quantization cost' — the model was never full precision — but 'does a from-scratch ternary model reach the quality bar for this task at a training budget you can afford, and do your inference kernels realize the theoretical bandwidth win on your target hardware?' Both halves must be true, and both are architectural properties you design for, not knobs you tune afterward.
The architecture: every piece explained
Top row: the layer that does everything. BitLinear is a drop-in replacement for nn.Linear; every linear projection in attention and the feed-forward blocks becomes a BitLinear, while embeddings, the final head, and normalization typically stay higher precision. Inside BitLinear the forward pass runs two quantizers. The weight quantizer uses an absmean scheme: compute the mean absolute value of the weight tensor as a scale λ, divide, then round each value to the nearest of {-1, 0, +1}. The zero level is what distinguishes b1.58 from earlier binary BitNet — allowing a weight to be exactly zero gives the network a way to prune connections implicitly and recovers most of the quality gap to full precision. The activation quantizer uses absmax per-token scaling to int8, so activations retain eight bits of dynamic range while weights are ternary.
Right of those, the ternary matmul is the payoff: with int8 activations and ternary weights, the inner product is a sum of int8 values added or subtracted according to weight sign, accumulated in int32, then multiplied once by the product of the weight scale and activation scale to return to real units. No per-element floating multiply appears in the hot loop. The per-tensor scales (kept in fp16) are the only floating-point survivors, applied at the tensor boundary rather than inside the accumulation.
Middle row: what makes training stable. The straight-through estimator keeps a full-precision master weight; the forward pass uses the quantized value, but the backward pass passes gradients through as if the rounding were the identity, so the optimizer updates the master copy and the ternary projection tracks it. A LayerNorm placed before quantization stabilizes the activation distribution so the absmax scale does not swing wildly batch to batch — without it, training diverges early. The packed weights store roughly 1.58 bits per parameter; in practice values are packed several to a byte (for example five ternary values encode into an 8-bit code, or simpler 2-bit-per-value packing trades density for kernel simplicity).
Bottom rows: turning bits into speed. The kernel is where theory meets silicon — either a bit-serial GEMV that processes ternary weights as sign masks, or a lookup approach that precomputes partial sums, or on modern CPUs a routine built around the widest available integer SIMD. The whole point is the memory-bandwidth win: streaming ~1.5GB instead of ~14GB per forward pass. The ops strip names what actually has to be true in production: the model is trained from scratch in the scheme, evaluation shows parity against the intended baseline, and — critically — the fast custom kernel produces outputs that match a slow reference implementation to numerical tolerance, because a subtly wrong kernel is a silent quality regression.
End-to-end flow
Trace a single decode step through a BitNet b1.58 model serving on a laptop CPU. A token id arrives; the embedding table (kept in higher precision) produces a hidden vector. It enters the first transformer block and hits a LayerNorm, which centers and scales the activation so the downstream absmax quantizer sees a well-behaved distribution.
Now the attention query projection, a BitLinear. At load time its weights were already ternarized and packed, with the per-tensor scale λ stored alongside; nothing is quantized at runtime on the weight side. The activation quantizer takes the normalized hidden vector, finds its per-token absmax, and maps it to int8. The kernel then computes the projection as signed accumulation: for each output neuron it walks the packed ternary weights, adding the int8 activation where the weight is +1, subtracting where -1, skipping zeros, accumulating into int32. A single fp16 multiply by λ_weight × λ_activation returns the result to real units. The same pattern runs for key, value, and output projections, and for the two large feed-forward matmuls that dominate the parameter count.
The performance character is visible here. On the memory side, the weights being read are a tenth the bytes, so the CPU's cache and memory system deliver them far faster; on the compute side, the inner loop is integer add/subtract, which even a modest core executes at high throughput with low power. The activations moving between layers are int8 or the normalized floats around the norms — small. The net effect is that a model which would be unusably slow in bf16 on this CPU generates several tokens per second, and the same math on a mobile NPU's integer datapath is faster still because the hardware was built for exactly this shape of arithmetic.
It is worth being precise about where floating point survives, because that is what keeps the scheme accurate rather than merely fast. The accumulation is pure integer — int8 activations summed and subtracted into an int32 register — but the two per-tensor scales (λ for the weights, derived from their absmean, and the activation's absmax scale) are fp16, and they are applied exactly once, at the boundary where the int32 accumulator is converted back to a real-valued activation. That single multiply per output element is cheap and it is where the network's dynamic range is restored. Getting it wrong in either direction is a bug: fold the scales in too early and you lose the integer-only inner loop that made the layer fast; size the accumulator too small and long inner-product sums overflow silently, corrupting outputs only at large sequence lengths where the sums are biggest. The design deliberately confines floating point to the tensor edges so the hot path stays integer while the arithmetic stays faithful.
Now the stress case that reveals the architecture's constraints. Suppose the custom kernel packs five ternary values into a byte but the reference implementation used a different bit ordering, and a boundary case near a packed-code edge is decoded with the wrong sign. Nothing crashes; the model simply produces slightly worse tokens, and only careful eval catches it — perplexity drifts up a few tenths, a benchmark drops two points, and there is no exception to trace. This is why BitNet operations lean so hard on reference-kernel validation: the failure mode of low-bit inference is not a crash but a quiet degradation that looks like 'the model just isn't as good as the paper claimed.' Catching it means running the fast kernel and the slow, obviously-correct one on the same inputs and asserting they agree before trusting any speedup.