Architecture
Custom quantization (GGUF). Kernels: CPU AVX2/AVX512, CUDA, Metal, Vulkan, ROCm. All backends supported.
Advertisement
Speed
Apple M-series: Metal backend, competitive with CUDA on similar VRAM. Modern iPhones run 3B models tolerably.
Advertisement
Deployment forms
Server mode (HTTP + OpenAI API). CLI (interactive chat). Library (embed in app). Mobile bindings.