Why architecture matters here
SLM deployment architecture matters because on-device AI is one bad OTA update away from bricking user experience. Signing, staged rollout, telemetry, rollback — all more important than cloud deploys.
Cost is upfront distribution + telemetry infra. On-device inference is user's hardware; that's cheap for you.
Reliability comes from staged rollout + rollback. Never global deploy without staging.
The architecture: every layer explained
Walk the diagram top to bottom.
Model artifact. Quantized (INT4 usually) + signed with provider key.
Distribution. OTA update or app bundle (Store review overhead).
Device runtime. llama.cpp, MLC, Core ML, or ONNX Runtime.
Version rollout. 1% canary → 5% → 25% → 100%. Monitor metrics per stage.
Fallback to cloud. For queries the on-device SLM handles poorly.
Telemetry. Privacy-preserving aggregate metrics (differential privacy).
Battery + thermal. Monitor; adapt (pause generation if hot).
Model cache. Load once at app start; keep in memory.
Update signing. Only signed models load. Prevents rogue OTA.
Rollback path. Bad update → revert to previous known-good.
End-to-end SLM deployment flow
Trace a deployment. New SLM v2 built and signed.
Canary: 1% of app installs get v2 on next launch. App downloads OTA; verifies signature; caches.
Telemetry aggregates over 24 hours: latency, tokens/sec, satisfaction proxy, thermal impact. Compared to v1 baseline.
v2 shows 15% faster inference; satisfaction stable; no thermal regression. Promote to 5%.
Repeat monitor. Ramp to 25% then 100% over a week.
Alternative: telemetry shows increased "escalated to cloud" rate. Investigate: v2 refusing more. Rollback: canary users' next launch reverts to v1.
User request that's too hard: app runs on-device SLM, gets low-confidence result; falls back to cloud. Cloud response returned. On-device still handles 85% of queries.
Thermal event: sustained generation heats device. App detects; pauses or slows to prevent throttling. User informed subtly.