Why architecture matters here

On-device fails on device diversity, battery drain, and update pipeline gaps. Architecture matters because runtime + accel + battery + update compose.

Advertisement

The architecture: every piece explained

The top strip is core. SLM checkpoint. Memory-mapped load. Runtime. NPU / GPU.

The middle row is management. Battery + thermal. Cache. Fallback to cloud. OTA updates.

The lower rows are ops. Privacy. Metrics. Ops — device tiering + eval + support.

On-device SLM — quantized + mmap + battery + CoreML/NNAPI + fallbackrun models where the user isSLM checkpointquantized weightsMemory-mapped loadno full RAMRuntimellama.cpp / ONNX / CoreML / NNAPINPU / GPUaccelerated execBattery + thermalthrottleCacheprompt + responseFallback to cloudcomplex queriesOTA updatessigned model swapPrivacydata stays localMetricsopt-in telemetryOps — device tiering + eval + supportguardhitrouteswapprotectmeasuremeasureoperateoperate
On-device SLM stack: quantized weights + runtime + fallback.
Advertisement

End-to-end flow

End-to-end: user query. Cache hit skips model. Miss: SLM runs on NPU; battery-throttle if hot. Complex query falls back to cloud with anonymized request. OTA updates the model when connected + charging.