Why architecture matters here
SLM edge quantization architecture matters because device constraints are real. A phone has 4-8GB usable RAM shared with the OS + apps; a laptop has 16-32GB. Fit or fail.
Cost is fixed at deploy time. Right format + runtime combination means a 3B model runs on a phone at usable speed; wrong combination means it doesn't run at all.
Reliability under thermal throttling is where careful design shows. Sustained inference heats the device; throttling kicks in; user experience degrades. Batching, pacing, and idle behavior all matter.
The architecture: every layer explained
Walk the diagram top to bottom.
SLM in FP16. The base — Phi-3.5-mini, Llama 3.2 3B, Qwen 3 3B. Foundation to quantize.
Quantizer. Tool chain converting to on-device format. llama.cpp's convert + quantize; MLC's compilation; coremltools for Apple.
Format Choice. INT4 (Q4_K_M in GGUF, standard); INT2 (Q2_K, extreme compression, quality risk); FP8 (limited edge support).
GGUF (llama.cpp). Cross-platform; Windows/Linux/macOS/Android/iOS. Flexible; widely supported. Q4_K_M is standard default.
MLC (mobile). Compiles model with TVM per target device. Best perf on Android/iOS with GPU/NPU.
Core ML (iOS/macOS). Apple's on-device ML. Integrates with Neural Engine + GPU. Best iOS integration.
Weight sharing. K-means clustering weights to a small codebook. Extra compression beyond bit-depth.
KV quantization. On-device long-context needs INT8/INT4 KV cache to fit.
Device Constraints. RAM budget (2-6GB for model); thermal throttling; battery drain rate.
Quality Eval. Task-specific benchmarks; language coverage; edge-case handling.
End-to-end deployment flow
Trace a deployment. Target: Phi-3.5-mini (3.8B) on iOS.
Download FP16 weights from Hugging Face. Convert to Core ML format via coremltools: quantize INT4 per-block, target Neural Engine + GPU compute.
Resulting Core ML package: ~2GB. Fits in app bundle or download-on-first-launch.
Alternative: llama.cpp GGUF Q4_K_M. ~2.3GB. Runs on iOS via a Metal-backed llama.cpp fork or as a library.
Benchmark on iPhone 15 Pro: Core ML gets 30 tokens/sec; llama.cpp gets 22. Core ML wins on Apple silicon.
Quality eval on Phi-3 benchmarks: MMLU 68 vs FP16 baseline 70 (acceptable), HumanEval 55 vs 58. Translation to target languages tested.
Deployment: bundle Core ML in the app. Cold start ~1s to load model. First-token latency ~200ms after warm.
Sustained use: thermal throttles after 5 minutes of continuous generation. App backs off with idle detection.
Battery: 20% drain per hour of active generation. Documented in app.