Why architecture matters here
The forcing function on device is not latency alone but the trio of power, thermal, and memory, all fixed by the hardware in the user's hand. A datacenter can add a GPU; a phone cannot. So every architectural choice is about staying inside an envelope: the model must be small enough to fit in the memory budget alongside the OS and other apps, quantized enough that its arithmetic runs on the NPU's integer units, and structured so that sustained inference doesn't push the SoC past its thermal limit and force a clock reduction. These constraints interact — a bigger model needs more memory bandwidth, which burns more power, which generates more heat, which triggers throttling, which misses the latency target. The SLM-on-NPU architecture is the discipline of keeping all four in balance at once.
Operator coverage is the first make-or-break property, and it is invisible in a naive benchmark. The NPU accelerates a specific set of operations — the matmuls, common activations, normalization patterns it was built for. If the model contains an op the NPU doesn't support (an unusual attention variant, a custom normalization, an unsupported data layout), the compiler must cut the graph there, run that op on the CPU, and move the tensors back and forth across the memory bus. A single unsupported op in the middle of the transformer block, executed every layer every token, can dominate runtime and power even though it is a tiny fraction of the FLOPs — the cost is the fallback and the data movement, not the math. This is why 'does it run on the NPU' is really 'what fraction of it runs on the NPU, and where are the seams.'
Quantization is the second. The NPU wants int8 or int4 weights and int8 activations; the model was trained in bf16. Post-training quantization or quantization-aware training brings it into that format, but low-bit quantization can cost quality, and the cost is task-dependent — a model that keeps its perplexity may still regress on the specific reasoning your product needs. The architecture forces a joint decision: the numeric format is chosen for what the NPU accelerates, and the quality has to be validated against the product's tasks in that exact format, because there is no runtime fallback to higher precision.
The third property is the prefill/decode asymmetry, which determines how the hardware is stressed. Prefill — processing the prompt — is a big parallel matmul, compute-bound, where the NPU's MAC array and the GPU both shine. Decode — generating one token at a time — reads the entire weight set and the growing KV cache per token, so it is memory-bandwidth bound, and the NPU's on-chip SRAM and the device's DRAM bandwidth set the ceiling. Understanding which phase you are optimizing tells you whether to spend effort on compute scheduling or on memory layout and KV-cache management, and whether to route prefill to the GPU while decode runs on the NPU.
The architecture: every piece explained
Top row: from model to silicon. The model graph arrives already quantized and with ops fused where possible (a matmul + bias + activation collapsed into one NPU-friendly operation). The compiler or delegate — the vendor's runtime that sits under a framework like an on-device inference API — walks the graph and partitions it: contiguous runs of supported ops become NPU subgraphs dispatched to the accelerator, and unsupported ops are left for CPU or GPU. The quality of this partitioning is decisive; a good delegate keeps long unbroken NPU runs, a bad match shatters the graph into many small subgraphs with expensive handoffs between them. The NPU MAC array is the engine: a systolic or dataflow grid of integer multiply-accumulate units that streams a tile of weights and activations through, producing partial sums, at very high ops-per-watt. The on-chip SRAM holds the tiles it is actively working on — weights, activation blocks, and slices of the KV cache — so the array stays fed without hitting DRAM every cycle.
Middle row: the supporting cast. DMA and the memory subsystem stream weights from DRAM into SRAM ahead of the MAC array's need; because decode re-reads all weights per token, DMA bandwidth and the SRAM working set largely determine decode speed. CPU fallback catches ops the NPU can't do — correct but slow, and costly because of the data movement to and from NPU memory. The GPU path is often used for prefill, whose parallelism suits the GPU, or as an alternative accelerator for ops the NPU lacks; heterogeneous scheduling decides which engine runs what. Power and thermal management is not a footnote — it is a first-class constraint that caps sustained clocks, so the SoC's ability to hold a frequency under a warm chassis sets the real, sustained throughput, distinct from the burst number a cold-start benchmark reports.
Bottom rows: the memory story that dominates decode. The KV cache grows by one entry per layer per generated token; on device, where memory is scarce, its size and the bandwidth to read it every step are a central constraint, which is why on-device SLMs lean on KV quantization, sliding-window attention, and small context budgets. The prefill vs decode split is the scheduling axis: prefill is compute-bound (route to the array/GPU, batch the prompt), decode is bandwidth-bound (minimize weight and KV re-reads, keep the working set in SRAM). The ops strip is the shipping checklist: verify what fraction of ops actually landed on the NPU, validate the quantized model's quality on real tasks, profile sustained (not burst) performance under thermal load, and budget memory so the model plus KV cache plus the rest of the app coexist.
End-to-end flow
Trace a single on-device interaction. A user types a prompt into an app that embeds a quantized 2B-parameter SLM. At install or first run, the vendor delegate compiled the model graph for this specific SoC: it partitioned the transformer blocks into NPU subgraphs, identified one unsupported normalization variant, and assigned it to the CPU, and it laid out weights for the NPU's preferred tiling.
Prefill runs first. The prompt's tokens are processed together — a large, parallel matmul workload. The runtime routes this to the compute-strong path: the NPU's MAC array (and on some devices the GPU) chews through the attention and feed-forward projections in a handful of big operations, building the initial KV cache for every layer. This phase is compute-bound and finishes quickly relative to its FLOP count because the hardware is built for exactly this shape. The one CPU-bound normalization op runs each layer, adding a small but real per-layer tax and a tensor round-trip across the memory bus.
Decode then generates the response token by token. Each token requires reading the entire weight set and the current KV cache through the NPU. This is bandwidth-bound: the MAC array is mostly waiting on DMA to stream weights from DRAM into SRAM, and the KV cache — which grew during prefill and grows by one entry per layer each token — must be read every step. The on-chip SRAM holds as much of the active working set as it can; the delegate tiles the matmuls to maximize SRAM reuse and minimize DRAM traffic. Tokens stream out at the device's sustainable rate, and the CPU fallback op fires once per layer per token, its data movement now a recurring cost on the critical path.
The stress case is thermal and memory pressure together. The user holds a long conversation. As decode continues, the SoC warms; past a threshold the thermal governor drops the clock, and token rate falls — the model that did twelve tokens per second on a cold device now does seven, and the UI feels sluggish even though nothing 'failed.' Meanwhile the KV cache has grown with the conversation, and combined with the model weights it is pressing on the memory budget; another app in the background is killed, or the runtime must evict and recompute context. This is the edge-inference reality the benchmark hid: sustained performance under thermal load and a growing KV cache, not the first-token burst. It is why on-device SLM work spends so much energy on KV quantization, context-window discipline, sustained-clock profiling, and keeping every hot op on the NPU — because each of those directly moves the sustained number the user actually experiences.