Why architecture matters here
SLM edge deployment matters because network round-trips destroy the user experience for interactive features and destroy the economics for high-volume ones. A voice assistant that requires a 300 ms cloud hop for every query feels sluggish; the same query on-device in 20 ms feels responsive. A code completion that costs a fraction of a cent per suggestion is fine; a hundred completions per developer per day at cloud pricing multiplies.
Privacy is a second driver. On-device inference keeps user prompts and completions on the user's hardware. For regulated data — health, legal, personal — this is often the difference between a viable product and a blocked one. Apple Intelligence, Samsung Galaxy AI, and Microsoft Recall (post-redesign) all leaned into this argument.
Reliability is the third. Cloud-only assistants stop working when the network is patchy. On-device SLMs work in the subway, on a plane, in low-connectivity environments. Users notice; they trust the feature more.
The architecture: every piece explained
Walk the diagram top to bottom.
Base SLM. A pretrained small model — Llama 3.2 1B/3B, Phi 3.5, Gemma 2 2B, Qwen 2.5 3B, Mistral 3B. Choose based on license, quality-per-parameter, and framework support. This is your starting point.
Distillation / QLoRA. Adapting the base to your domain. Distillation from a larger teacher model produces higher quality for the same size. QLoRA adds LoRA adapters and quantizes; efficient and effective for domain adaptation.
Quantized Artifact. The final on-device format. INT4 groupwise (GGUF for llama.cpp, MLC for MLC-LLM) or FP8. Model + tokenizer + inference config bundled. Signed and versioned.
Edge Runtime. llama.cpp (cross-platform, C++), MLC (mobile-focused, WASM support), Apple Core ML (integrated into iOS/macOS), ONNX Runtime (broad platform), TensorRT (NVIDIA). Choose based on target platform.
Device Constraints. RAM (2-8 GB total; models eat a chunk), thermal budget (sustained inference heats the device), battery (inference costs power). Design for the worst-case device you must support.
Local Inference. On-device tokens per second depends on hardware. Modern phones do 15-30 tps for 3B INT4 models; laptops do 50-150 tps; specialized accelerators (Apple Neural Engine, Qualcomm Hexagon) more.
Cloud Fallback. For queries the local SLM cannot handle — long context, complex reasoning, tools the SLM does not have. The fallback API is scoped and rate-limited.
Router (local vs cloud). The decision layer. Based on query complexity, model confidence, latency budget, and user preference, route locally or escalate to cloud. Router quality determines the ratio of local vs cloud calls.
Telemetry. Privacy-preserving. Aggregate metrics only (token/s, success rate); no user content leaves the device. Differential privacy for sensitive aggregates.
OTA Model Update. Signed model deltas ship over the air. Rollback path if a new model regresses. Version discipline; test on a canary device fleet before global rollout.
Governance. On-device policy (what the SLM can and cannot do). Cloud audit for the fallback queries (where user consent may allow). Separate policies for edge vs cloud paths.
End-to-end query flow
Trace a query. A user asks their phone's assistant, "summarize this article for me." The article is 3,000 words.
The router estimates: this is summarization, well within the SLM's capability; input fits in the 8k context of the on-device model; latency target is 2 seconds. Route local.
The on-device SLM (Phi 3.5, INT4, 3.8 GB in RAM) processes the article. Prefill takes 800 ms; decode streams the summary at 20 tokens per second. The summary appears in 2 seconds. No network required.
Same user asks: "how does this article relate to the paper I read last week? Compare their conclusions." This is more complex; the local model would struggle. The router escalates.
The query goes to the cloud API with user consent (a settings choice). The cloud model handles the comparison. Result streams back; 1.2 second first token, 40 tps thereafter.
Telemetry records the routing decision, local latency, cloud latency, and success. Aggregated across the fleet, product managers see: 87% of queries handled locally, 13% escalated. Average local latency 800 ms; cloud 1.4 s. Cost per query drops 60% compared to cloud-only.