Small Language Model Edge Deployment Architecture in Depth

The budget: memory, bandwidth and heat

Start with memory, because it decides what can load at all. Weight memory is parameters times bits per weight divided by eight. A 3.2-billion-parameter model at 16 bits needs about 6.4 GB, which no phone app will be allowed to hold. At 4-bit group-wise quantization the effective rate is closer to 4.5 bits per weight once per-group scales are counted, so about 1.8 GB. That is feasible on a device with 8 GB of RAM shared with the OS and other apps.

Then add the KV cache, which grows with context. Per token it stores a key and a value for every layer and every KV head:

kv_bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element

# Llama 3.2 3B: 28 layers, 8 KV heads (grouped-query attention), head_dim 128
2 * 28 * 8 * 128 * 2 bytes (fp16)  = 114,688 bytes  ~= 112 KiB per token
4,096-token context                ~= 448 MiB
same cache quantized to 8-bit      ~= 224 MiB

Speed comes next. Generating each token during decode reads essentially all the weights once, so decode is bound by memory bandwidth, not arithmetic. A useful ceiling is effective bandwidth divided by bytes read per token. A phone sustaining, say, 50 GB/s of real bandwidth against 1.8 GB of weights tops out near 28 tokens per second before any overhead. Prefill, which processes the prompt, is the opposite: it is batched over many tokens and limited by compute, which is where NPUs and GPUs earn their keep. Thermal limits and battery then lower both numbers under sustained use. KV cache and NPU acceleration go deeper on each side.

The build pipeline: from base model to signed artifact

The build pipeline runs in the cloud and produces one artifact per device tier. It has five stages:

1. Choose the base. Pick by licence, tokenizer, quality per parameter on your task, and whether your target runtime supports the architecture. Families such as Llama 3.2, Gemma, Phi and Qwen all ship 1-4B variants. Benchmarks on generic leaderboards are a weak signal; build a task eval first.
2. Adapt. Distill from a larger teacher on your task distribution, or fine-tune with LoRA. Small models gain more from narrow adaptation than large ones do, because they have less general capacity to fall back on.
3. Quantize. Weight-only 4-bit with per-group scales (methods such as AWQ, GPTQ or llama.cpp's K-quants) is the usual default. Keep embeddings and the output head at higher precision if quality drops. Calibrate on in-domain text, not generic web text.
4. Convert. Export to the runtime's format with the exact tokenizer and chat template the model was trained with. A template mismatch is a silent, common quality bug.
5. Evaluate on devices. Re-run the task eval on the quantized artifact, on real hardware, and record quality, time to first token, tokens per second, peak memory and temperature after five minutes of sustained use.

Each artifact ships with a manifest: a content hash, a signature, the minimum device tier, the context length it was validated at and the tasks it passed. The device trusts the manifest, not the file name. SLM quantization compares the quantization methods.

Choosing a runtime

The runtime decides which accelerators you can reach and which formats you ship. There is no universal winner, and many products ship two.

There is also a build-versus-borrow choice. Apple and Google both expose OS-managed on-device models to apps on supported devices: Apple through its Foundation Models framework, Android through Gemini Nano. Using the platform model costs you no download and no memory budget of your own, but you give up control of the model, its version and its update schedule, and coverage is limited to recent hardware. Bring your own model when you need a specific fine-tune, a stable version or cross-platform behaviour. GGUF and llama.cpp explains the most portable option in detail.

On the device: tiering and routing

Your fleet is not one device. Define a few tiers by memory, accelerator and OS version, and at startup pick the largest artifact the device can actually hold. Measure free memory at runtime. Do not trust the spec sheet. Mobile operating systems kill apps that grow too large, and a killed app is worse than a smaller model.

TIERS = [  # largest first; numbers come from your device-farm measurements
    Tier("3b-q4", weights=1.8e9, layers=28, kv_heads=8, head_dim=128, ctx=4096, overhead=0.3e9),
    Tier("1b-q4", weights=0.8e9, layers=16, kv_heads=8, head_dim=64,  ctx=4096, overhead=0.2e9),
]

def kv_bytes(t, ctx, elem_bytes=2):
    return 2 * t.layers * t.kv_heads * t.head_dim * elem_bytes * ctx

def pick_tier(device):
    budget = device.available_memory() * 0.5       # leave headroom for the OS and the app
    for t in TIERS:
        need = t.weights + kv_bytes(t, t.ctx) + t.overhead
        if need <= budget and device.runtime_supports(t.name):
            return t
    return None                                    # cloud-only, or feature hidden

def route(req, tier, device, policy):
    if tier is None or req.task not in tier.validated_tasks:
        return cloud_if_allowed(req, policy)
    if count_tokens(req.prompt) + req.max_new_tokens > tier.ctx:
        return cloud_if_allowed(req, policy)       # or chunk and summarise locally
    if device.thermal_state() >= SERIOUS and policy.cloud_allowed(req):
        return CLOUD
    return LOCAL

def cloud_if_allowed(req, policy):
    return CLOUD if policy.cloud_allowed(req) else REFUSE_WITH_REASON

Two details matter. The router checks the task against what this artifact was validated for, not what the model might manage. And policy.cloud_allowed encodes the user's consent and the data class, so content the user marked private is refused rather than silently escalated.

Shipping models: OTA updates and rollback

Models change more often than apps, so ship them separately from the app binary. Store two slots. Download the new artifact into the inactive slot, verify its hash and signature against the manifest, load it once to run a short self-test with a fixed prompt and expected output shape, and only then flip the pointer. Keep the old slot until the new one has served real traffic without crashing. That gives you instant rollback without a download.

Roll out in stages: internal devices, then a small percentage per tier, watching crash rate, memory kills, latency and task success before widening. Do not expect binary deltas to save much. A fine-tune changes most weights, and re-quantization shifts nearly every byte, so a new 1.8 GB model is usually close to a full download. If you update often, keep the base model fixed and ship small LoRA adapters instead. They are typically tens of megabytes, provided your runtime can apply adapters at load time. Download over unmetered connections when charging, and never block the feature on an update.

Worked example: on-device document summaries

A notes app wants on-device summaries of 3,000-word documents, about 4,000 tokens of input, on phones with 8 GB of RAM. The memory budget for tier 3b-q4 is 1.8 GB of weights plus 448 MiB of fp16 KV cache at 4,096 tokens plus about 0.3 GB of runtime buffers, roughly 2.6 GB in total. The 50 percent headroom rule in pick_tier therefore needs about 5.2 GB reported free, which many 8 GB phones do not show. So the device farm measures free memory per model, and only models that reliably report enough get the 3B tier. The rest fall back to 1b-q4, which has its own validated task list. A 4,000-token prompt plus a 250-token summary overflows a 4,096 context, so the router either chunks the document or escalates. The team chooses map-reduce: summarise two halves locally, then summarise the summaries.

Latency follows from the measurements. Suppose the device farm shows prefill at 400 tokens per second on the NPU and decode at 18 tokens per second; treat these as example figures, not promises. The two map passes prefill about 4,000 tokens in total (about 10 s) and generate two 150-token partial summaries (about 17 s). The reduce pass adds a short prefill and a 200-token summary (about 12 s). That is roughly 40 s, which is too slow for an interactive tap but fine as a background job that notifies when done. The team ships it that way, streams the final pass so the user sees progress, and escalates to the cloud only when the user asks for an immediate result and has allowed cloud processing. The arithmetic, not the demo, decided the UX.

Telemetry, governance and failure modes

Telemetry has to tell you whether the feature works without collecting what users wrote. Record counts and distributions only: route chosen, time to first token, tokens per second, peak memory, thermal state, fallback reason, app kills during inference and an explicit thumbs-up or thumbs-down rate. Aggregate on the device before upload, and apply differential privacy where small cohorts could identify someone.

Governance has two halves. On the device, the system prompt, the tool allow-list and any output filters ship inside the signed manifest, so they cannot be edited locally. On the cloud path, log and retain fallback requests under the same policy as any other server-side processing, and only with consent. Keep a kill switch per artifact version, so a model found to misbehave can be disabled remotely in minutes.

The common failure modes are app kills from memory growth (watch the KV cache at long contexts); thermal throttling that halves decode speed after a few minutes; quality loss from quantization on specific skills such as arithmetic or less-represented languages, which aggregate benchmarks hide; tokenizer or chat-template mismatches after conversion; slow cold starts that memory-mapping the weights mitigates; and a cloud fallback rate that creeps up until the cost case disappears. Distillation for production helps when the local quality gap is the cause.

What to do next

1. Write the task eval first, with real examples, and run it on the full-precision and quantized models.
2. Compute the memory budget (weights plus KV cache plus overhead) for each candidate at your context length.
3. Estimate the decode ceiling from bandwidth, then measure time to first token, tokens per second and sustained temperature on your lowest supported device.
4. Choose a runtime per platform, or decide that the OS-provided model is enough.
5. Implement tiering by measured free memory and a router with explicit validated tasks and consent rules.
6. Ship models in signed, two-slot OTA packages with staged rollout, self-test and a kill switch.
7. Instrument content-free telemetry and set alert thresholds for fallback rate, app kills and thermal throttling.