Why architecture matters here
SLM serving is about constraints. On device: RAM, storage, battery, thermal. At the edge: hardware variance, network unreliability. The architecture matters because you must balance latency, quality, cost, and battery.
Fallback to cloud rescues quality when the SLM is insufficient. Response cache saves cycles for common queries. Guardrails prevent harmful outputs even offline.
With the pieces mapped, you can ship SLM-powered features that feel snappy and stay compliant.
The architecture: every piece explained
The top strip is the request path. Client request is text or speech. Edge / device runtime — ONNX Runtime, ggml/llama.cpp, TFLite — hosts the model. Quantized model uses int4 or int8 weights for memory savings. Response cache serves hot prompt hits without inference.
The middle row is efficiency + safety. Micro-batching batches multiple concurrent requests on-device when possible. Fallback to cloud kicks in for complex queries or SLM low-confidence outputs. Guardrails run on-device to prevent harmful output. Telemetry — opt-in — feeds improvements without violating privacy.
The lower rows are lifecycle. Model update ships over-the-air with signed packages and rollback. Battery / thermal budget throttles usage to protect device. Ops covers feature parity with cloud, A/B testing, and user opt-in for features.
End-to-end flow
End-to-end: user asks a question on a mobile app. Cache miss. Edge runtime loads int4 quantized 3B model. Response generated in 400 ms. Guardrails clear. Response returned. Later user asks a complex reasoning question; SLM's confidence low; fallback to cloud with anonymized request. Total offline queries handled: 89%. Battery impact minimal thanks to thermal throttling. Model update pushed OTA next release; if regression detected, rollback within one release cycle.