All 72 articles, sorted alphabetically
Alpaca Format, in depth: the instruction, input and output record, the prompt template it implies, prompt-masked loss, validation and moving to chat templates
What the Alpaca fine-tuning format really is: the three-field record from Stanford Alpaca, the two prompt templates it is trained with, how prompt tok…
Read article →Anthropic Fine-Tuning Format, in depth: the Claude JSONL record on Amazon Bedrock, its rules and quotas, validation, conversion and portability
The training-data format for fine-tuning Claude on Amazon Bedrock: the system-plus-messages JSONL record, the role and key rules, how it differs from …
Read article →AWQ + GPTQ Quantization, in depth: 4-bit weights for 1-4B models, where the bytes really go, why small models are fragile and a calibrate, serve and evaluate workflow
A practical guide to AWQ and GPTQ for small language models: what W4A16 means and why it speeds up decoding, both algorithms explained from first prin…
Read article →ChatML Format, in depth: the token layout, templates as code, tool turns, adding it to a base model, and closing the turn-forgery hole
A practical guide to ChatML, the im_start and im_end chat format used by Qwen and many open-weight models: why the markers must be single special toke…
Read article →Dataset Ordering, in depth: shuffling that actually mixes, curricula that earn their keep, and an order you can reproduce
How training data order affects small language models: why SGD needs well-mixed batches, two-level shard and buffer shuffling, epochs across ranks, le…
Read article →Distillation for Production Inference, in depth: sizing the student to the serving budget, draft models, cascades and safe rollout
How to use knowledge distillation to cut production inference cost: start from the serving budget, choose a student shape by memory and KV-cache arith…
Read article →DPO, in depth: a practitioner's workflow for small models, from building preference pairs and the loss in code to TRL configuration, reading the training metrics and evaluating the result
How to run Direct Preference Optimization on a small language model end to end: what the loss does, a from-scratch PyTorch implementation, a worked nu…
Read article →Evaluating SLMs, in depth: what each public benchmark measures, which still separate small models, how many items you need, and the practical tests that decide
A guide to benchmarking small language models: what MMLU, MMLU-Pro, GPQA, GSM8K, MATH, IFEval, BBH, HellaSwag, ARC, HumanEval and function-calling sui…
Read article →Fine-Tuning Phi-4, in depth: choosing phi-4 or Phi-4-mini, fused LoRA targets, chat formats, the pad/EOS trap and a training loop
How to fine-tune Microsoft's Phi-4 family: the verified architecture of phi-4 (14B) and Phi-4-mini (3.8B), the fused projection layers LoRA must …
Read article →Fine-Tuning Operations Pipeline, in depth: manifests, data snapshots and leakage checks, cacheable DAG steps, gates against production, registry promotion and retrain triggers
How to run fine-tuning of small language models as a repeatable pipeline rather than a one-off job: a manifest that pins base model, tokenizer, data a…
Read article →Gemma Family, in depth: from Gemma 1 to Gemma 4, hybrid attention and KV memory, per-layer embeddings, the 26B mixture of experts, prompt format and LoRA fine-tuning
Google's Gemma open models in depth: the release lineage from Gemma 1 to Gemma 4, the Gemma 4 checkpoints, how sliding-window and global attentio…
Read article →GGUF format, in depth: the bytes on disk, metadata keys, tensor layout, quantization blocks and how to build, inspect and split a model file
A first-principles guide to GGUF, the single-file model format used by llama.cpp and the ggml ecosystem: header, typed metadata, tensor info and align…
Read article →JSONL Format for Fine-Tuning, in depth: line framing, the record shapes trainers expect, streaming writers, validation, splits and the loader traps
How to build JSONL fine-tuning data that trains what you think it trains: the three JSON Lines rules, standard and conversational record shapes for SF…
Read article →Llama Chat Template: Llama 2 [INST] and Llama 3 Format
Llama chat template explained: Llama 2's [INST] tags, Llama 3's header role tokens, apply_chat_template, system prompts, and use in vLLM and…
Read article →llama.cpp, in depth: ggml and its backends, one request through llama-server, memory budgets you can compute, offload and batching flags, measuring speed and quality, and running it in production
How llama.cpp works and how to run it well: the ggml, libllama and tools layers, one chat request traced through slots, prompt processing, continuous …
Read article →LoRA, in depth: implementing low-rank adapters in code, checking they trained, and managing them as artifacts
LoRA from the code up: a from-scratch adapter layer with merge and unmerge, injecting it by module name, what zero-initialised B means on the first st…
Read article →MCP for Edge Devices, in depth: stateless MCP on gateways and on-device models, transports, tool budgets and safety
How to use the Model Context Protocol at the edge with revision 2026-07-28: which tier speaks MCP, why per-request metadata suits devices that restart…
Read article →Ollama, in depth: the request path, memory and context planning, concurrency, Modelfiles, the API and running it for a team
How Ollama serves local models: what the server, scheduler and runner processes do, how to budget weights and KV cache before you pull a model, why co…
Read article →On-Device LLM Inference, in depth: the engine inside the app, from cold load to streaming, context limits, saved state and surviving the OS
How to run an LLM inside a mobile or desktop app: loading and warm-up, a prefill and decode loop on a worker, streaming and cancellation, context-wind…
Read article →On-Device SLMs, in depth: Gemini Nano as a system model in Android AICore and Chrome, availability states, quotas, hybrid fallback and prompt design
How to build on a small language model that the platform owns rather than one you ship: Gemini Nano in Android's AICore and in Chrome's buil…
Read article →OpenAI fine-tuning data format, in depth: messages JSONL, weights, tool calls, DPO pairs, RFT graders, validation and portability
The OpenAI fine-tuning data contract explained: the messages JSONL format for supervised fine-tuning, per-message weights, tool-calling examples, DPO …
Read article →PEFT, in depth: choosing between LoRA, DoRA, IA3, prompt tuning, prefix tuning and adapters for a small language model, with exact parameter counts and runnable configs
Parameter-efficient fine-tuning for small language models: where LoRA, DoRA, IA3, prompt tuning, prefix tuning, bottleneck adapters and BitFit attach …
Read article →Phi Family, in depth: the textbook-data thesis, every generation from phi-1 to Phi-4-reasoning-vision, and how to choose and run one
A practical guide to Microsoft's Phi small language models: the synthetic textbook-quality data thesis behind them, the lineage from phi-1 to Phi…
Read article →Phi, Qwen, Gemma: Small Models Compared, in depth: licences, real memory footprints, architectures, prompt formats and a bake-off you can run
How to choose between Microsoft Phi, Alibaba Qwen and Google Gemma small models: current sizes and licences checked against the model cards, effective…
Read article →Preference Data Collection, in depth: prompt sourcing, candidate generation, the judging interface, quality control, AI feedback and releasing a dataset you can trust
How to collect preference pairs for reward models and DPO: what a preference record contains, sourcing prompts, generating on-policy candidates, pairw…
Read article →QLoRA from the storage up: NF4 built from scratch in NumPy, checked against bitsandbytes, block size, outliers, double quantization and auditing a loaded model
A hands-on look at what QLoRA stores and computes: deriving the NF4 codebook and matching bitsandbytes to 1e-7, blockwise quantization and nibble pack…
Read article →Qwen Family, in depth: five generations, the specialist branches, hybrid linear attention, thinking switches, serving flags and how to pick a checkpoint
A working guide to Alibaba's Qwen open-weight models: the lineage from Qwen2.5 through Qwen3, Qwen3-Next, Qwen3.5 and Qwen3.6, the Coder, VL, Omn…
Read article →RLHF Pipeline, in depth: SFT, preference data, reward model and PPO for small language models, with the gates between them
An end-to-end guide to running RLHF on a small language model: the four artifacts and the gates between them, collecting on-policy preference data, tr…
Read article →SFT, in depth: what supervised fine-tuning optimises, chat templates, loss masks, packing, loss normalisation and teaching a model to stop
Supervised fine-tuning from first principles: the objective, why the chat template must match inference, assistant-only labels built from character of…
Read article →ShareGPT Format, in depth: multi-turn records, role normalisation, tool turns and per-turn loss masking
How the ShareGPT conversation format stores multi-turn chat data for fine-tuning: the from/value record, mapping roles to a chat template, validation …
Read article →SLM batching -- throughput for small-model serving
Deep-dive on SLM batching: the per-request inefficiency, static vs continuous batching (join/leave per decode step), the throughput-latency tradeoff, …
Read article →SLM distillation architecture
Deep-dive on small-language-model distillation: teacher forward, KD loss, temperature, curriculum, data augmentation, eval, and deployment.
Read article →SLM Distillation Data Recipe, in depth
A concrete recipe for building a sequence-level distillation dataset for a small language model: task spec, prompt pool, k-sample teacher generation, …
Read article →SLM Distillation from an LLM, in depth: teacher access, logit caching, tokenizer mismatch and a memory-safe KD loss
The teacher-side engineering of distilling a large language model into a small one: white-box, top-k and text-only teacher access, online versus offli…
Read article →DPO alignment architecture for small language models
Deep-dive on Direct Preference Optimization for SLMs: preference-pair pipelines, frozen reference models and precomputed log-probs, the beta dial, LoR…
Read article →SLM early-exit inference architecture
Deep-dive on early-exit (adaptive-depth) inference for small language models: intermediate exit classifiers and confidence thresholds, shared LM heads…
Read article →Small Language Model Edge Deployment Architecture in Depth
SLM edge deployment architecture in depth: memory and bandwidth budgets from first principles, the build pipeline, runtimes, device tiering and routin…
Read article →SLM Evaluation Architecture in Depth: Task Suites, Judges, Regression Gates and On-Device Checks
How to build an evaluation system for small language models: dataset registry and slices, pinned run configuration, programmatic scorers, calibrated m…
Read article →Evaluating Small Models: Common Pitfalls, in depth: scoring formulation, templates, extraction, contamination and the artifact you ship
The concrete mechanisms that make small language model scores wrong, each with a reproduction and a fix: cloze versus multiple-choice scoring and leng…
Read article →SLM Fine-Tuning, in depth: choosing between full fine-tuning, adapters, continued pretraining and DPO, with the memory math and training loops for each
A practical guide to fine-tuning small language models: when fine-tuning is the right tool, continued pretraining versus SFT versus DPO preference tun…
Read article →FlashAttention on-device
Deep-dive on FlashAttention for on-device small language models, the algorithmic reframing that computes attention without ever materializing the quad…
Read article →Small Models for Tool Calling, in depth: choosing a base model, chat templates, training data, LoRA fine-tuning and evaluation
How to make a small language model reliable at tool calling from the model side: what the task decomposes into, choosing a base model, chat templates …
Read article →Training a Small Language Model from Scratch: Architecture Choices and the Full Pipeline, with Pseudocode
An engineering plan for pretraining a 100M-1B parameter language model: sizing from a compute budget, architecture choices that matter at small scale,…
Read article →SLM function calling architecture, in depth: tool registry, retrieval, constrained decoding, validation, authorization and escalation
How to build a dependable function-calling harness around a small language model: a tool registry with hidden metadata, retrieval to shrink the tool l…
Read article →GGUF runtime architecture
Deep-dive on the GGUF runtime behind llama.cpp and on-device inference: a single self-describing file holding metadata, tokenizer, and block-quantized…
Read article →Guided decoding architecture for SLMs
Deep-dive on constrained/structured generation for small models: compiling JSON schema, regex, and CFG to a finite-state automaton, per-step logit mas…
Read article →SLM Inference Optimization, in depth: finding the bottleneck with bytes-per-token arithmetic, then applying the right technique in the right order
A method for making small language models run faster: measure time to first token and time per output token, locate the bottleneck with roofline and b…
Read article →KV Cache for Small LLMs: Paging, Prefix Sharing, Quantization
KV cache architecture for small LLM serving: paged blocks, prefix sharing, int8/fp8 KV cache quantization, and swapping paused sequences out to host m…
Read article →SLM Landscape 2026, in depth: the open small-model families, the architecture shifts that change deployment, and how to shortlist
A working map of open small language models as of October 2026: Phi-4-mini, Qwen3.5, Gemma 4, Ministral 3, SmolLM3, Granite 4.0 and Llama 3.2 by size …
Read article →LoRA, in depth: fine-tuning a small language model with low-rank adapters, from dataset to merged edge build
A practical LoRA workflow for small language models: why small hidden sizes change the choices, a worked parameter and memory budget for Llama-3.2-3B,…
Read article →Multi-LoRA serving architecture
Deep-dive on multi-LoRA inference serving: the low-rank adapter decomposition and why the base GEMM is shareable, the adapter registry and GPU-residen…
Read article →Small MoE Architecture, in depth: fine-grained experts, balancing, upcycling and running on a device
Mixture-of-experts at small scale, where memory rather than compute is the constraint: fine-grained and shared experts, bias-based load balancing, spa…
Read article →On-device SLM NPU acceleration architecture
Deep-dive on running small language models on mobile NPUs: graph compilation and partitioning, the integer MAC array and on-chip SRAM, quantization fo…
Read article →SLMs on Mobile in 2026, in depth: OS-provided models versus bundled models, memory and thermal budgets, delivery, routing and on-device benchmarking
An app engineer's guide to shipping small language models on phones in 2026: why decode speed is bounded by memory bandwidth, Apple's Founda…
Read article →Small Language Models Explained: What They Are, How They Are Built, and Where They Beat Large Models
An overview of small language models: how to define small by deployment rather than a parameter cutoff, why over-training, curated data and distillati…
Read article →Prefix caching for SLM serving
Deep-dive on prefix caching for small-language-model serving: block-hashing token prefixes, a radix index of cached KV blocks, paged and reference-cou…
Read article →Pretraining Data for Small Language Models: Filtering, Deduplication, and Mixture Design, with Pseudocode
How to build the pretraining corpus for a small language model: stage ordering, per-document provenance and drop reasons, exact and near deduplication…
Read article →The SLM Pretraining Loop in Pseudocode: Optimizer, LR Schedule, Mixed Precision, Checkpoints, and Loss-Spike Recovery
The small-language-model pretraining loop as one system: order of operations in a step, AdamW parameter groups, schedules as pure functions of step, b…
Read article →SLM pruning -- removing weights to shrink the model
Deep-dive on SLM pruning: removing unimportant weights, unstructured (sparse) vs structured (dense) pruning, the crucial sparsity-vs-hardware question…
Read article →QLoRA, in depth: when a 4-bit base is worth it for small language models, and how to ship the result
QLoRA for small language models: how a frozen NF4 base and bf16 LoRA adapters fit together, a memory ledger for 0.5B to 8B models showing why embeddin…
Read article →SLM Edge Quantization Architecture in Depth
A 2500-word walkthrough of SLM edge quantization: GGUF, MLC, Core ML, INT4/Q4_K_M formats, weight sharing, KV quantization, device constraints, qualit…
Read article →SLM tensor parallelism architecture
Deep-dive on tensor parallelism for serving small language models: column- and row-parallel sharding of attention and MLP, two all-reduces per layer, …
Read article →Vocabulary trimming architecture
Deep-dive on tokenizer vocabulary trimming for small language models: why V x d dominates small-model budgets, corpus surveys, keep sets and byte-fall…
Read article →SLMs in IoT: Giving 'Dumb' Appliances a Voice with Local 1B Parameter Models
The promise of the "smart home" and the Internet of Things (IoT) has often been undermined by a critical dependency: the cloud.
Read article →Small Models for Classification, in depth: encoders vs small decoders, LLM-labelled training data, calibration and cascades
How to replace an LLM classifier with a small model: choosing between a fine-tuned encoder such as ModernBERT, a decoder SLM with a classification hea…
Read article →Structured Output with Small Models, in depth: schema design, constrained decoding, validation, repair and field-level evaluation
How to get reliable JSON out of 1B to 8B models: why small models fail at structure, when to prompt, constrain or fine-tune, how to design schemas a s…
Read article →Synthetic Training Data Generation, in depth: designing input diversity, Self-Instruct and Evol-Instruct, grounding, filtering and avoiding model collapse for small models
How to generate synthetic training data for a small language model: why diversity is the hard part, seed taxonomies crossed with personas and difficul…
Read article →The Economics of SLMs: Why Startups Are Saving Millions by Switching to Smaller Footprints
The promise of Large Language Models (LLMs) like GPT-4 is undeniably alluring, offering unparalleled general intelligence, complex reasoning, and creative…
Read article →TinyLlama and the 1B Frontier: What Can You Actually Do with a 1-Billion Parameter Model?
While headlines often celebrate the latest Large Language Models (LLMs) boasting hundreds of billions or even trillions of parameters, a quiet revolution is…
Read article →Training Data Curation Pipeline, in depth: turning tickets, logs and expert answers into a versioned fine-tuning dataset for a small model, with lineage, dedup, quality checks and leak-free splits
How to build the curation pipeline that produces fine-tuning data for a small language model from your own sources: a content-addressed record and dro…
Read article →vLLM, in depth: serving small language models with prefix caching, multi-LoRA, structured outputs and speculation
How to serve small language models with vLLM: why a 1-8B model shifts the bottleneck to per-step host work and concurrency, how the V1 scheduler spend…
Read article →WebMCP: Exposing Website Tools to AI Agents in the Browser, with Code
How WebMCP lets a web page register JavaScript tools with JSON schemas for browser agents: the current draft API (document.modelContext, registerTool,…
Read article →