SLM Edge Deployment — Running Models on Phones and Devices

Why it matters

Edge deployment enables offline AI, privacy, and latency wins. It's a distinct engineering challenge from cloud deployment. Learning the tooling opens up new products.

The architecture

Model conversion: PyTorch → GGUF (llama.cpp), ONNX (cross-platform), Core ML (Apple), CoreML/NNAPI (Android).

Quantization: INT4 or lower typical. GGUF supports mixed precisions per layer.

How it works end to end

Runtime selection: llama.cpp for broad hardware, ONNX Runtime for enterprise cross-platform, MLC LLM for mobile GPU acceleration.

Hardware: modern phones have NPUs (Apple Neural Engine, Qualcomm Hexagon). Use them for major speedup.

Memory footprint: 3-7B quantized fits in 2-4 GB RAM. Careful for older devices.