All 24 articles, sorted alphabetically
Llama Chat Template
Llama 2's [INST] tags and Llama 3's role tokens. How chat templates shape model behavior, message structure, system prompts,…
Read article →SLM batching -- throughput for small-model serving
Deep-dive on SLM batching: the per-request inefficiency, static vs continuous batching (join/leave per decode step), the throughput-latency tradeoff, …
Read article →SLM distillation architecture
Deep-dive on small-language-model distillation: teacher forward, KD loss, temperature, curriculum, data augmentation, eval, and deployment.
Read article →SLM Distillation Architecture in Depth
A 2500-word walkthrough of SLM distillation: teacher, task data, student, response/logit/reasoning distillation, diverse sampling, filtering, evaluati…
Read article →DPO alignment architecture for small language models
Deep-dive on Direct Preference Optimization for SLMs: preference-pair pipelines, frozen reference models and precomputed log-probs, the beta dial, LoR…
Read article →SLM early-exit inference architecture
Deep-dive on early-exit (adaptive-depth) inference for small language models: intermediate exit classifiers and confidence thresholds, shared LM heads…
Read article →Small Language Model Edge Deployment Architecture in Depth
A 2500-word walkthrough of SLM edge deployment: base model, distillation, quantization, edge runtime, router, cloud fallback, telemetry, OTA, and gove…
Read article →FlashAttention on-device
Deep-dive on FlashAttention for on-device small language models, the algorithmic reframing that computes attention without ever materializing the quad…
Read article →SLM function calling architecture
Deep-dive on reliable tool use from small language models: schema registries compiled to decoding grammars, streaming validation, repair loops, fallba…
Read article →GGUF runtime architecture
Deep-dive on the GGUF runtime behind llama.cpp and on-device inference: a single self-describing file holding metadata, tokenizer, and block-quantized…
Read article →Guided decoding architecture for SLMs
Deep-dive on constrained/structured generation for small models: compiling JSON schema, regex, and CFG to a finite-state automaton, per-step logit mas…
Read article →SLM KV cache architecture
Deep-dive on the KV cache in small-language-model serving: why decode is memory-bandwidth-bound, prefill vs decode, paged KV blocks and block tables (…
Read article →Multi-LoRA serving architecture
Deep-dive on multi-LoRA inference serving: the low-rank adapter decomposition and why the base GEMM is shareable, the adapter registry and GPU-residen…
Read article →On-device SLM NPU acceleration architecture
Deep-dive on running small language models on mobile NPUs: graph compilation and partitioning, the integer MAC array and on-chip SRAM, quantization fo…
Read article →PagedAttention architecture
Deep-dive on PagedAttention for SLM serving: fixed-size KV blocks mapped through per-sequence block tables, on-demand allocation with an O(1) free-lis…
Read article →Prefix caching for SLM serving
Deep-dive on prefix caching for small-language-model serving: block-hashing token prefixes, a radix index of cached KV blocks, paged and reference-cou…
Read article →SLM pruning -- removing weights to shrink the model
Deep-dive on SLM pruning: removing unimportant weights, unstructured (sparse) vs structured (dense) pruning, the crucial sparsity-vs-hardware question…
Read article →SLM Edge Quantization Architecture in Depth
A 2500-word walkthrough of SLM edge quantization: GGUF, MLC, Core ML, INT4/Q4_K_M formats, weight sharing, KV quantization, device constraints, qualit…
Read article →Speculative decoding
Deep-dive on speculative decoding: draft-verify mechanism, parallel verification exploiting the memory-bound insight, acceptance rate and speedup, exa…
Read article →SLM tensor parallelism architecture
Deep-dive on tensor parallelism for serving small language models: column- and row-parallel sharding of attention and MLP, two all-reduces per layer, …
Read article →Vocabulary trimming architecture
Deep-dive on tokenizer vocabulary trimming for small language models: why V x d dominates small-model budgets, corpus surveys, keep sets and byte-fall…
Read article →SLMs in IoT: Giving 'Dumb' Appliances a Voice with Local 1B Parameter Models
The promise of the "smart home" and the Internet of Things (IoT) has often been undermined by a critical dependency: the cloud.
Read article →The Economics of SLMs: Why Startups Are Saving Millions by Switching to Smaller Footprints
The promise of Large Language Models (LLMs) like GPT-4 is undeniably alluring, offering unparalleled general intelligence, complex reasoning, and creative…
Read article →TinyLlama and the 1B Frontier: What Can You Actually Do with a 1-Billion Parameter Model?
While headlines often celebrate the latest Large Language Models (LLMs) boasting hundreds of billions or even trillions of parameters, a quiet revolution is…
Read article →