Edge Inference for LLM, in depth: bandwidth-bound decode, memory and KV budgets, runtimes, thermals and a hybrid local-cloud router

Why decode on the edge is a bandwidth problem

An autoregressive model produces one token per forward pass, and at batch size one every pass must read every weight from DRAM. A matrix-vector product does about two floating-point operations per weight it reads, so the arithmetic units sit idle waiting for memory. The ceiling on decode speed is therefore simple: tokens per second is at most memory bandwidth divided by the bytes read per token, which is roughly the size of the quantised weights plus the KV cache being read. The per-operator version of this ledger is worked through in the decode compute math article; here we only need the headline.

Prefill, processing the prompt, is different. All prompt tokens go through each weight together, so it is a matrix-matrix product and compute-bound. On edge parts this is where the GPU or NPU earns its place: a long prompt on CPU alone can take many seconds before the first token appears, while decode speed changes far less than prefill does between CPU and GPU on the same memory bus. That is the single most useful fact in edge inference: offload helps time-to-first-token far more than it helps tokens per second, unless the GPU has its own faster memory.

Sizing weights, cache and speed

Size the deployment in three numbers: weight bytes, KV cache bytes and the bandwidth ceiling. Weight bytes are parameters times bits per weight over eight. The quantisation formats in the GGUF ecosystem have exact layouts for the simple types: Q8_0 stores blocks of 32 int8 weights plus one fp16 scale, 8.5 bits per weight; Q4_0 stores 32 four-bit weights plus a scale, 4.5 bits per weight. The popular k-quant mix Q4_K_M lands at about 4.8 to 4.9 bits per weight because it keeps some tensors at higher precision. KV cache bytes per token are 2 (keys and values) times layers times KV heads times head dimension times bytes per element.

def edge_budget(params_b, bits_per_weight, layers, kv_heads, head_dim,
                ctx_tokens, kv_bytes=2.0, bandwidth_gbs=100.0):
    weights_gb = params_b * 1e9 * bits_per_weight / 8 / 1e9
    kv_per_tok = 2 * layers * kv_heads * head_dim * kv_bytes          # bytes
    kv_gb = kv_per_tok * ctx_tokens / 1e9
    # every decode step reads all weights plus the live part of the cache
    ceiling_tps = bandwidth_gbs / (weights_gb + kv_gb)
    return dict(weights_gb=round(weights_gb, 2), kv_gb=round(kv_gb, 2),
                total_gb=round(weights_gb + kv_gb, 2), ceiling_tps=round(ceiling_tps, 1))

# 8B model, 32 layers, 8 KV heads, head_dim 128 (Llama 3.1 8B shape), Q4_K_M, 8K context
print(edge_budget(8.0, 4.85, 32, 8, 128, 8192, bandwidth_gbs=102))
# {'weights_gb': 4.85, 'kv_gb': 1.07, 'total_gb': 5.92, 'ceiling_tps': 17.2}

Real decode speed usually reaches only part of that ceiling, because kernels do not saturate the bus and attention, sampling and synchronisation add overhead; treat roughly half to three quarters of the ceiling as a planning range and replace it with a measurement as soon as you have the device. The ceiling uses a full 8K cache, the worst case at the end of a long conversation; early in a chat the cache is small and speed is closer to the weights-only number.

Worked example: ceilings across devices

These are arithmetic ceilings from the vendors' bandwidth figures with about 4.85 GB of weights for an 8B model and about 1.95 GB for a 3B model, not benchmark results. Two lessons follow. First, an 8B model on an 8 GB shared-memory board leaves almost nothing for the OS, the cache and your application, so the 3B class is the realistic choice there. Second, moving from 4-bit to 8-bit weights nearly halves decode speed on every row, which is why edge deployments quantise aggressively and spend quality budget elsewhere.

Choosing and measuring a runtime

The runtime decides which kernels run, how memory is mapped and what quantisation formats you can use. llama.cpp runs GGUF models on CPU, CUDA, Metal, Vulkan and other backends, memory-maps the model file and lets you choose how many layers to place on the GPU. MLX is Apple's array framework for Apple silicon, with its own quantised formats and the mlx-lm package; see MLX in depth. On Jetson, NVIDIA's TensorRT-LLM and the CUDA backend of llama.cpp are the usual choices. Phone and PC NPUs need vendor compilers and their own operator support, covered in edge NPUs in depth.

Measure before you choose. llama.cpp ships a benchmark tool that reports prefill and decode speed separately, which is the split that matters:

# prefill 512 tokens, decode 128 tokens, all layers on the GPU, flash attention on
llama-bench -m qwen2.5-3b-instruct-q4_k_m.gguf -p 512 -n 128 -ngl 99 -fa 1

# serve it with an 8K context and an 8-bit KV cache to halve cache memory
llama-server -m qwen2.5-3b-instruct-q4_k_m.gguf -c 8192 -ngl 99 -fa on \
    --cache-type-k q8_0 --cache-type-v q8_0

Flag syntax drifts between llama.cpp releases (recent builds take on, off or auto for flash attention), so check --help on the build you ship. Quantising the V cache needs flash attention enabled. For practical VRAM budgeting with a wrapper around the same engine, see Ollama on the GPU.

Techniques that move the numbers

Beyond quantisation, four techniques move the numbers. Speculative decoding uses a small draft model to propose several tokens that the main model verifies in one pass; because decode is bandwidth-bound, verifying several tokens costs nearly the same as generating one, so a good draft can raise tokens per second substantially, at the cost of memory for the draft model. The mechanics are in speculative decoding in depth.

KV cache quantisation to 8 bits halves cache memory with small quality loss on most tasks, which matters more on the edge than in the data centre because the cache competes with the weights for the same pool. Prompt caching keeps the KV state of a fixed system prompt so every request skips that part of prefill. Finally, choose context length on purpose: an 8K window that is never used still reserves memory in most runtimes.

Thermals and sustained performance

Phones and fanless boxes cannot sustain peak clocks. The first benchmark run on a cool device is the best number you will ever see; ten minutes into a conversation the SoC has heated, clocks have dropped and both prefill and decode slow down. Battery-powered devices may also cap power when the battery is low. Benchmark sustained performance: run a realistic conversation loop for at least ten minutes and plot tokens per second over time, with the device in its real enclosure and ambient temperature.

On Jetson, power modes are set with nvpmodel, jetson_clocks pins clocks at their maximum for the selected mode, and tegrastats reports utilisation, frequencies and temperatures; record them alongside your token rates. The general clock-management loop is explained in GPU thermal management. Design the product so that a throttled device is still acceptable, for example by capping response length or switching to a smaller model when the thermal state is serious.

A hybrid local-or-cloud router

Most products end up hybrid: a small local model handles the common, private, short requests and a cloud model handles the rest. The router is ordinary code that should be explicit and logged, not a hidden heuristic.

from dataclasses import dataclass

@dataclass
class DeviceState:
    free_mem_gb: float
    thermal: str            # "nominal", "fair", "serious", "critical"
    on_battery: bool
    battery_pct: int
    online: bool

def route(prompt_tokens, needs_private, task, dev: DeviceState, local_ctx=8192):
    if needs_private and not dev.online:
        return "local"                       # nothing else is allowed or possible
    if needs_private:
        return "local" if prompt_tokens < local_ctx else "refuse_or_summarise"
    if not dev.online:
        return "local" if prompt_tokens < local_ctx else "queue"
    if task in {"long_report", "complex_code", "multi_doc_reasoning"}:
        return "cloud"                       # quality gap too large for a 3B model
    if dev.thermal in {"serious", "critical"}:
        return "cloud"
    if dev.on_battery and dev.battery_pct < 20:
        return "cloud"
    if prompt_tokens > 2000 or dev.free_mem_gb < 1.0:
        return "cloud"                       # long prefill or memory pressure
    return "local"

Log every decision with its reason, the token counts and the observed latency. That log tells you where the local model is good enough and lets you tune thresholds with data instead of guesses. Keep the privacy rule first and unconditional: a request marked private must never fall through to the cloud branch because a later rule fired.

Operating a fleet of on-device models

Shipping a model to thousands of devices is a software distribution problem with very large files. Version the model, the runtime and the prompt template together, because a template mismatch quietly degrades quality. Sign model files and verify the hash before loading. Download in resumable chunks over unmetered networks, keep the previous model until the new one passes a local smoke test, and roll out in stages with a kill switch. Collect opt-in, privacy-preserving telemetry on load failures, out-of-memory kills, time-to-first-token and decode speed by device model, since the same build behaves very differently across hardware tiers.

Evaluate quality per quantisation level on your own task set, not only on public benchmarks. A 4-bit model can lose more on structured output, arithmetic or a minority language than its average scores suggest.

Failure modes

• Out-of-memory kills. Mobile operating systems terminate apps that exceed their budget; leave headroom and fall back to a smaller model or shorter context instead of crashing.
• Slow first token. Long prompts on CPU-only paths; offload prefill to the GPU or NPU and cache the system prompt.
• Throttling cliffs. Speed halves after minutes of use; test sustained runs and degrade gracefully.
• Template drift. The chat template in the app no longer matches the model version, producing rambling or broken output.
• Quantisation regressions. A lower-bit file passes generic tests but breaks JSON output or a language your users need.
• Unverified model files. A tampered or truncated download loaded without a signature check.

What to do next

1. Write down your target device tiers with memory and published bandwidth, and run the budget function for each candidate model and context length.
2. Run llama-bench or your runtime's equivalent for prefill and decode on the real device, then a ten-minute sustained loop with temperatures logged.
3. Pick the quantisation level using your own evaluation set, including structured output and every language you support.
4. Quantise the KV cache and cap context length to what the product actually needs.
5. Implement an explicit, logged local-or-cloud router with the privacy rule first.
6. Set up signed, staged model delivery with rollback and opt-in telemetry by device model.