Small Language Models

Alpaca Format

instruction + input + output triple. Simple + widely-adopted.

Anthropic Fine-Tuning Format

AWS Bedrock + Anthropic Console. Similar chat JSONL, Claude-specific fields.

AWQ + GPTQ Quantization

GPU-optimized 4-bit quantization. Serving standard for OSS models.

ChatML Format

Special tokens delimit roles. Standard for chat model tokenization.

Dataset Ordering

How training order affects learning. Shuffling defaults + curriculum learning.

Distillation for Production Inference

Teacher-student in 2026: when distillation is worth it.

DPO

Preference pairs → policy. Simpler than PPO. Modern RLHF replacement.

Evaluating SLMs

MMLU + GSM8K + HumanEval + chatbot arena. Choose per use case.

Fine-Tuning Phi-4

QLoRA recipe for narrow tasks.

Fine-Tuning Operations Pipeline

Data → train → eval → deploy → monitor. Production ML workflow for LLMs.

Gemma Family

Gemma 1/2/3. Built on Gemini research. Open weights for commercial use.

GGUF Format

Model + tokenizer + metadata in one file. Cross-platform. Standard for CPU + Apple + edge.

JSONL Format for Fine-Tuning

Newline-delimited JSON: one training example per line. Universal format.

Llama Chat Template

Llama 2's [INST]/[/INST] tags. Llama 3's newer role tokens.

llama.cpp

C++ inference. Foundation of Ollama, LM Studio, Jan. Runs on any hardware.

LoRA

Freeze base weights, train small rank-r matrices. 10-100x fewer trainable params.

MCP for Edge Devices

Run MCP servers on IoT + mobile. Sensor + local file + control tool exposure.

Model Deployment at Edge

ONNX Runtime, MLC, Executorch, Core ML. Cross-platform edge inference.

Ollama

One-command LLM install + run. Foundation of local LLM stack.

On-Device LLM Inference

What runs on phones, laptops, and edge boxes in 2026.

On-Device SLMs

3B models baked into OS. Privacy + latency + battery matter.

OpenAI Fine-Tuning Data Format

Chat format JSONL with weight parameter. Function calling + tools supported.

PEFT

Umbrella: LoRA, adapters, prompt tuning, prefix tuning. Compare + choose.

Phi Family

Phi-1 through Phi-4. 'Textbooks are all you need' philosophy.

Phi, Qwen, Gemma: Small Models Compared

The 1B-9B parameter sweet spot in 2026.

Preference Data Collection

Ranked pairs for DPO/RLHF. Human + LLM annotators. Real workflow.

QLoRA

4-bit quantized base + LoRA adapters. Fine-tune 65B on single GPU.

Qwen Family

Strong Chinese + multilingual. 0.5B to 72B. Growing worldwide.

RLHF Pipeline

Full RLHF: supervised fine-tune, reward model, PPO. ChatGPT's original recipe.

SFT

Fine-tune base model on instruction/response pairs. Foundation of instruct models.

ShareGPT Format

Conversation list with role tags. Common for Vicuna-style training.

SLM batching -- throughput for small-model serving

Deep-dive on SLM batching: the per-request inefficiency, static vs continuous batching (join/leave per decode step), the throughput-latency tradeoff, …

SLM Deployment Architecture in Depth

A 2500-word walkthrough of SLM deployment: signed artifact, distribution, device runtime, canary rollout, cloud fallback, telemetry, thermal, rollback…

SLM Edge Deployment

How to deploy SLMs on mobile, embedded, and edge hardware using ONNX, GGUF, and specialized runtimes.

SLM distillation architecture

Deep-dive on small-language-model distillation: teacher forward, KD loss, temperature, curriculum, data augmentation, eval, and deployment.

Knowledge Distillation

How distillation transfers capability from a teacher model to a student, and why distilled models often beat directly-trained equivalents.

SLM Distillation Architecture in Depth

A 2500-word walkthrough of SLM distillation: teacher, task data, student, response/logit/reasoning distillation, diverse sampling, filtering, evaluati…

SLM Distillation Data Recipe

Teacher outputs filtering and CoT capture.

SLM distillation from LLM architecture

Deep-dive on SLM distillation from LLM: teacher, prompts, rationale outputs, student SFT/KD/RLHF, task distribution, eval.

DPO alignment architecture for small language models

Deep-dive on Direct Preference Optimization for SLMs: preference-pair pipelines, frozen reference models and precomputed log-probs, the beta dial, LoR…

SLM early-exit inference architecture

Deep-dive on early-exit (adaptive-depth) inference for small language models: intermediate exit classifiers and confidence thresholds, shared LM heads…

Small Language Model Edge Deployment Architecture in Depth

A 2500-word walkthrough of SLM edge deployment: base model, distillation, quantization, edge runtime, router, cloud fallback, telemetry, OTA, and gove…

SLM Evaluation

How to evaluate SLMs: benchmarks (MMLU, GSM8K, HumanEval), task-specific eval, and calibrating against larger models.

SLM evaluation architecture

Deep-dive on SLM evaluation: benchmark suite, task-specific tests, human eval, safety, regression, latency + cost, CI integration.

Evaluating Small Models: Common Pitfalls

Why your small model looks worse than it is.

SLM Fine-Tuning

How to fine-tune SLMs effectively: data quality, hyperparameters, evaluation, and when fine-tuning beats prompting.

FlashAttention on-device

Deep-dive on FlashAttention for on-device small language models, the algorithmic reframing that computes attention without ever materializing the quad…

Small Models for Tool Calling

When a 3B model beats GPT-4 for the agent's tool layer.

SLM function calling architecture

Deep-dive on reliable tool use from small language models: schema registries compiled to decoding grammars, streaming validation, repair loops, fallba…

GGUF runtime architecture

Deep-dive on the GGUF runtime behind llama.cpp and on-device inference: a single self-describing file holding metadata, tokenizer, and block-quantized…

Guided decoding architecture for SLMs

Deep-dive on constrained/structured generation for small models: compiling JSON schema, regex, and CFG to a finite-state automaton, per-step logit mas…

SLM Inference Optimization

Techniques beyond quantization: batching, speculative decoding, flash attention, and continuous batching for SLM serving.

SLM KV cache architecture

Deep-dive on the KV cache in small-language-model serving: why decode is memory-bandwidth-bound, prefill vs decode, paged KV blocks and block tables (…

SLM Landscape 2026

State of small language models. Under 15B params. Runs on laptops + phones.

LoRA

How LoRA fine-tunes large models by learning small low-rank updates to weight matrices, saving memory and enabling many task adapters.

Multi-LoRA serving architecture

Deep-dive on multi-LoRA inference serving: the low-rank adapter decomposition and why the base GEMM is shareable, the adapter registry and GPU-residen…

Small MoE architecture

Deep-dive on small MoE for SLMs: base, expert count, top-K routing, load balance loss, distillation, serving fit, metrics.

On-device SLM NPU acceleration architecture

Deep-dive on running small language models on mobile NPUs: graph compilation and partitioning, the integer MAC array and on-chip SRAM, quantization fo…

SLMs on Mobile in 2026

iOS Foundation Models and Android Gemini Nano.

On-device SLM architecture

Deep-dive on on-device SLM: quantized checkpoints, memory-mapped load, runtimes, NPU/GPU, battery + thermal, cache, fallback, OTA.

Small Language Models

What SLMs are, why they matter, and how they compete with much larger models on many tasks.

PagedAttention architecture

Deep-dive on PagedAttention for SLM serving: fixed-size KV blocks mapped through per-sequence block tables, on-demand allocation with an O(1) free-lis…

Prefix caching for SLM serving

Deep-dive on prefix caching for small-language-model serving: block-hashing token prefixes, a radix index of cached KV blocks, paged and reference-cou…

Model Pruning

How pruning removes redundant weights or attention heads from models, and how sparsity accelerates inference.

SLM pruning -- removing weights to shrink the model

Deep-dive on SLM pruning: removing unimportant weights, unstructured (sparse) vs structured (dense) pruning, the crucial sparsity-vs-hardware question…

QLoRA

How QLoRA combines 4-bit quantization with LoRA to fine-tune 70B models on a single consumer GPU.

Model Quantization

How quantization reduces model size and inference cost by using lower-precision weights and activations.

SLM Edge Quantization Architecture in Depth

A 2500-word walkthrough of SLM edge quantization: GGUF, MLC, Core ML, INT4/Q4_K_M formats, weight sharing, KV quantization, device constraints, qualit…

SLM router architecture

Deep-dive on SLM router: intent classifier, model registry, cost + latency budget, fallback ladder, quality gate, streaming adapter.

SLM serving architecture

Deep-dive on SLM serving on device/edge: quantized model, response cache, micro-batching, cloud fallback, guardrails, update, device budgets.

Speculative decoding

Deep-dive on speculative decoding: draft-verify mechanism, parallel verification exploiting the memory-bound insight, acceptance rate and speedup, exa…

SLM tensor parallelism architecture

Deep-dive on tensor parallelism for serving small language models: column- and row-parallel sharding of attention and MLP, two all-reduces per layer, …

Vocabulary trimming architecture

Deep-dive on tokenizer vocabulary trimming for small language models: why V x d dominates small-model budgets, corpus surveys, keep sets and byte-fall…

SLMs in IoT: Giving 'Dumb' Appliances a Voice with Local 1B Parameter Models

Small Models for Classification

Cheap inference for high-volume routing.

Structured Output with Small Models

JSON schemas at 3B parameters.

Synthetic Training Data Generation

Use frontier LLM to generate SFT/DPO data. Common + effective when done right.

The Economics of SLMs: Why Startups Are Saving Millions by Switching to Smaller Footprints

TinyLlama and the 1B Frontier: What Can You Actually Do with a 1-Billion Parameter Model?

Training Data Curation Pipeline

Dedup, quality filter, tokenize. Foundation of good models.

vLLM

PagedAttention + continuous batching. 10x+ throughput vs naive.

WebMCP

MCP servers as browser extensions. Anthropic's browser agent enabling protocol.