All 382 articles, sorted alphabetically
The Autoregressive Generation Loop
How a language model actually produces text: the autoregressive factorization p(sequence) = prod p(x_t | x_<t), the two phases of inference (comput…
Read article →Backpropagation Through Transformers
How backpropagation and the chain rule train transformers: forward vs backward pass, the chain rule for composed functions, the vector-Jacobian-produc…
Read article →Beam Search Decoding: Algorithm, Length Penalty, Example
Beam search decoding for transformers: log-probability scores, beam width and expand-then-prune, a worked example, the length penalty, and when to sam…
Read article →The Complete Transformer Block
How a complete pre-norm transformer block composes attention, a feed-forward network, two residual connections, and two normalizations end to end: the…
Read article →CPU Cache Hierarchy and Transformer Inference
How the CPU memory hierarchy -- registers, L1, L2, L3, and DRAM -- decides small-language-model inference speed on a CPU. Latency and bandwidth for ea…
Read article →CPU LLM Inference: llama.cpp, ONNX Runtime, OpenVINO
How LLMs run on a CPU with llama.cpp, ONNX Runtime or OpenVINO: compute-bound prefill vs memory-bound decode, quantization, threads, NUMA, and a speed…
Read article →CPU Matmul Kernels and BLAS
How a fast CPU matmul (GEMM) is really built: why GEMM is the transformer workhorse, the naive triple loop and its cache-miss problem, loop reordering…
Read article →CPU Memory Budget for SLM Training
A full accounting of the CPU RAM budget for running a small language model: model weights (params x bytes by precision), the KV cache (2 x L x n_kv x …
Read article →Cross-Entropy Loss for Next-Token Prediction
Cross-entropy loss for language modeling from first principles: maximum likelihood and negative log-likelihood, the one-hot target and -log p(correct …
Read article →Dataloader and Tokenization Pipeline
How the data loading and tokenization pipeline feeds a language model: raw text to tokens, document packing with separators into fixed-length sequence…
Read article →The Geometry of Dot Products
The geometry of the dot product and why attention is built on it: the dot product as projection and as similarity, the a·b = |a||b|cos(theta) identity…
Read article →End-to-End CPU SLM Recipe
A single worked recipe that ties the whole series together: build and run a ~100M-parameter small language model destined for CPU inference. Pick an a…
Read article →Feed-Forward Network
A first-principles deep-dive on the position-wise feed-forward network (FFN/MLP) in a transformer block: the two linear layers with a 4x expansion fac…
Read article →Fine-Tuning Math
The math of fine-tuning: the memory ledger of full fine-tuning (weights + gradients + optimizer states = ~16 bytes/param for Adam in fp32), why that c…
Read article →FlashAttention
A first-principles walk through FlashAttention: the IO-aware, tiled attention algorithm, online (streaming) softmax with a running max and running nor…
Read article →Gradient Accumulation and Microbatches
How gradient accumulation simulates a large effective batch on limited memory by summing gradients over K microbatches before an optimizer step: the e…
Read article →Gradient Clipping and Training Stability
Gradient clipping and training stability from first principles: why gradients explode and loss spikes appear, global-norm clipping (scale g by min(1, …
Read article →KV Cache
The exact math of the transformer KV cache: why autoregressive decode would recompute every past key and value without it, what the cache stores, the …
Read article →What Is KV Cache Quantization? int8 vs int4 Bytes per Token
What KV cache quantization is: the bytes-per-token math, fp16 vs int8 vs int4 worked out, per-token vs per-channel scales, key outliers, GQA, paged at…
Read article →LayerNorm
The math of layer normalization: normalizing across the feature dimension per token, the mean and variance, (x - mu)/sqrt(sigma^2 + epsilon), the lear…
Read article →Learning Rate Schedules
A first-principles deep dive on learning-rate schedules for training transformers: why a constant learning rate fails, linear warmup and why it stabil…
Read article →Linear Algebra for Transformers
The linear algebra that underlies transformers, from first principles: vectors and matrices, matrix multiplication as the workhorse operation, dot pro…
Read article →Long Context Strategies
A taxonomy of long-context strategies with the complexity math for each: the two costs that explode with sequence length -- O(N^2) attention compute a…
Read article →Diagnosing Loss Curves
A practical guide to reading training and validation loss curves: the healthy power-law decay shape, learning-rate-too-high divergence and NaN spikes,…
Read article →Mixed Precision Training
A first-principles deep dive on mixed-precision training: the FP32, FP16, and BF16 bit layouts, the exponent-vs-mantissa (range-vs-precision) trade, w…
Read article →Mixture of Experts
The math of Mixture-of-Experts: replacing a dense FFN with N experts and a softmax router, top-k sparse activation, the decoupling of total parameters…
Read article →Multi-Head Attention Math: Heads, d_k, W_O, Params and FLOPs
The math of multi-head attention: splitting d_model into h heads of size d_k, per-head W_Q, W_K, W_V, concat and W_O, tensor shapes, parameter and FLO…
Read article →Multi-Token Prediction (MTP)
A deep dive on multi-token prediction (MTP): predicting several future tokens per position instead of one, multiple prediction heads sharing a trunk, …
Read article →Output Projection and Logits
How a transformer turns its final hidden state into next-token probabilities: the output projection (unembedding / LM head) mapping d_model to vocab, …
Read article →Perplexity in LLMs: exp(Cross-Entropy), Bits, Worked Example
What perplexity measures for language models: exp of cross-entropy in nats, the branching-factor intuition, a worked example, bits-per-byte, tokenizer…
Read article →Sinusoidal Positional Encoding Math: The 10000^(2i/d) Formula
Sinusoidal positional encoding from first principles: why attention is order-blind, the 10000^(2i/d) wavelengths, relative position as rotation.
Read article →Quantization Layouts in Memory
How quantized transformer weights are actually laid out in memory: the affine quantization map q = round(x/scale) + zero_point, symmetric vs asymmetri…
Read article →Residual Connections and Gradient Flow
The math of residual connections in transformers: the y = x + F(x) skip, why it creates a gradient highway (dL/dx = dL/dy (1 + F')), how the iden…
Read article →RLHF and DPO
The math of RLHF and DPO: preference data and the Bradley-Terry model, learning a reward model, the KL-constrained reward-maximization objective and i…
Read article →RMSNorm: Formula, Backward Pass, RMSNorm vs LayerNorm
RMSNorm explained: y = x / sqrt(mean(x^2) + eps) * gamma, why dropping mean-centering works, its savings vs LayerNorm, the backward pass, and a worked…
Read article →Extending RoPE to Longer Contexts
How to extend a RoPE model's context window: a brief recap of rotary position embedding (rotating Q/K by position-dependent angles), why naive ex…
Read article →Greedy vs Sampling in LLMs: Temperature, Top-p, Min-p Math
LLM decoding strategies worked through: softmax, greedy decoding, temperature scaling and entropy, top-k, top-p (nucleus) and min-p on one logit vecto…
Read article →Scaled Dot-Product Attention Explained: Q, K, V Worked Example
The scaled dot-product attention formula softmax(QK^T / sqrt(d_k))V step by step: Q, K, V projections, why scale by sqrt(d_k), causal masking, tiny ma…
Read article →SGD vs Adam vs AdamW Optimizer Math
A first-principles walk through optimizer math: plain SGD, momentum as an exponential moving average of gradients, RMSProp as an EMA of squared gradie…
Read article →SIMD Instructions for Transformer Math
How SIMD vectorization speeds up transformer kernels on the CPU: one instruction over many lanes, AVX2/AVX-512 (8/16 fp32 lanes) and ARM NEON, how the…
Read article →SLM Architectures
A deep dive comparing modern small language model architectures across the design axes that actually differentiate them: depth vs width, attention var…
Read article →Softmax Derivation: Formula, Max Subtraction, Jacobian
Softmax derived step by step: why the exponential, shift invariance and subtracting the max logit, temperature, the Jacobian, log-sum-exp, a worked ex…
Read article →Speculative Decoding Math: Exact Rejection Sampling and Speedup
Speculative decoding math: draft and verify, modified rejection sampling, the acceptance rate, expected tokens per cycle, the speedup formula, Medusa,…
Read article →The Future of CPU SLM in 2026 and Beyond
Why CPU-based small language models are becoming genuinely viable: sub-4-bit and QAT quantization, stronger small models from over-training and distil…
Read article →Tied Embeddings: Weight Tying in Transformers
Tied embeddings explained: sharing the input embedding with the output projection, the parameter savings, why it works, the softmax bottleneck, and AL…
Read article →3D Parallelism Explained: Data, Tensor and Pipeline Parallel
3D parallelism combines data, tensor and pipeline parallelism: N = DP x TP x PP, TP over NVLink in a node, PP across nodes, and a 4096-GPU worked exam…
Read article →Ablation Study Math
The math behind ablation studies in LLM research: treatment effects, compute- and parameter-matched controls, seed variance and standard error, confid…
Read article →Beyond ReLU
GeLU, SiLU, and SwiGLU compared from first principles: exact formulas and derivatives, why smooth and gated activations beat ReLU in transformers, the…
Read article →Activation Checkpointing: Recomputation to Cut Training Memory
Activation checkpointing (activation recomputation): save segment boundaries, recompute activations in backward, replay dropout RNG, the sqrt(N) trade…
Read article →Activation Memory Math
How to count activation memory in a transformer from first principles: which tensors autograd must save and why, the per-layer 34sbh + 5as^2b accounti…
Read article →Activation Patching
Activation patching from first principles: the clean/corrupted two-prompt setup, the intervention written as math, logit-difference and normalized-rec…
Read article →Activation Quantization
Why activations are harder to quantize than weights: per-input dynamic range, outlier channels and tokens, static vs dynamic quantization, per-tensor …
Read article →Activation Recomputation Math
The math of activation recomputation in transformer training: the 34sbh + 5as^2b activation-memory formula, the one-third FLOPs overhead of full recom…
Read article →Activation Scaling
How activation variance propagates through a transformer: the fan-in variance identity, Xavier and He init, why the residual stream accumulates varian…
Read article →Adafactor Optimizer: Factored Second Moments vs Adam Memory
The Adafactor optimizer: how a rank-1 factorization shrinks Adam second-moment memory to n+m numbers, update clipping, relative step size, a worked ex…
Read article →Adam vs AdamW vs SGD: The Optimizer That Trains Transformers
Adam and AdamW compared with SGD: momentum, per-parameter second-moment scaling, bias correction, decoupled weight decay in AdamW, and how to set lr a…
Read article →AdamW Math Deep Dive
The math of AdamW: why L2 regularization in the gradient and weight decay on the weights are the same thing under SGD but provably different under Ada…
Read article →Bottleneck Adapters Explained: PEFT Math, Houlsby vs Pfeiffer
The math of bottleneck adapters in PEFT: down-project, nonlinearity, up-project, near-identity init, Houlsby vs Pfeiffer, parameter counts, adapters v…
Read article →Adaptive RAG
Adaptive RAG explained from first principles: deciding whether to retrieve at all, routing queries by predicted complexity into no-retrieval, single-s…
Read article →Admission Control Math
The math of admission control for LLM serving: Little's law, M/M/1 latency blowup near saturation, why token-length variance makes LLM queues fra…
Read article →Agent Scaling Laws
How LLM agent capability scales: the geometric horizon model S(H) = p^H, why per-step error compounds, the H_50 task-horizon metric and its doubling t…
Read article →ALiBi Math: Head Slopes, Distance Penalty and a Worked Example
ALiBi (Attention with Linear Biases) explained: the -m*(i-j) score bias, per-head slopes, a worked example, why it extrapolates, and how it differs fr…
Read article →Alignment Tax Math
The alignment tax: the capability-benchmark regression that can accompany RLHF, why it happens (KL drift from the pretrained distribution, reward-mode…
Read article →Annoy
How Annoy (Approximate Nearest Neighbors Oh Yeah) works: random-projection binary trees that split space by the perpendicular bisector of two sampled …
Read article →MongoDB Atlas Vector Search
The math behind MongoDB Atlas Vector Search: why exact kNN is O(N*d), how the HNSW graph reaches sublinear search, what M, efConstruction and numCandi…
Read article →Attention Complexity: Why Attention Is O(N^2) in Context Length
Why self-attention costs O(N^2) compute and memory in sequence length: the QK^T matrix, attention vs FFN FLOPs, prefill vs decode, KV cache, FlashAtte…
Read article →Attention Math From First Principles: Query, Key, Value Lookup
How attention works, derived from a soft dictionary lookup: query, key and value projections, sqrt(d_k) scaling, softmax weights, shapes, and a worked…
Read article →Attention variants architecture
A comparative map of attention variants: multi-head, multi-query, grouped-query, multi-head latent, sliding-window, sparse and linear attention, score…
Read article →AWQ Math
The math of AWQ (Activation-aware Weight Quantization): why a small set of salient weight channels identified by activation magnitude dominate quality…
Read article →Backpressure Math
The math of backpressure in an LLM serving pipeline: queue depth as the integral of arrival-minus-service rate, Little's law applied stage by sta…
Read article →Backprop architecture
The systems architecture of the transformer backward pass: what the chain rule actually costs in memory and bandwidth, gradient accumulation as a batc…
Read article →Bandwidth-Bound Operations
Why autoregressive decoding is memory-bandwidth-bound, not compute-bound: the arithmetic intensity of GEMV, bytes-per-token accounting for weights and…
Read article →Batched GEMM Math: Strided-Batched and Grouped GEMM for MoE
Batched GEMM in transformers: per-head attention shapes, strided-batched strides, grouped GEMM for MoE, why small matmuls are memory-bound, worked FLO…
Read article →Batch Size Math for LLM Training
The math of batch size in LLM training: the gradient as a noisy estimator and its 1/B variance, the gradient noise scale and critical batch size, the …
Read article →Beam Search Math: Log-Prob Scoring and Length Normalization
The math of beam search in transformers: log-prob scoring, the expand-score-prune loop, length normalization, the beam search curse, KV-cache cost, vs…
Read article →BF16 (bfloat16) Training Explained: Bit Layout, BF16 vs FP16
What BF16 is and why it trains without loss scaling: 1 sign, 8 exponent and 7 mantissa bits, FP32 range, precision limits, FP32 master weights, BF16 v…
Read article →BGE Embeddings: How BAAI General Embedding and BGE-M3 Work
BGE embeddings: the bi-encoder, CLS vs mean pooling, cosine similarity, InfoNCE contrastive training, hard negatives, BGE-M3 dense and sparse modes, M…
Read article →Bio Capability Evals
How to measure the biology and bio-risk capabilities of large language models: the difference between knowledge probes and task-based evals, the uplif…
Read article →BitFit Theory
BitFit explained from first principles: freeze every weight matrix and train only the bias vectors. Which biases are updated, the exact parameter coun…
Read article →bitsandbytes int8 Quantization
How bitsandbytes does 8-bit: block-wise absmax quantization and why blocks bound outlier damage, the linear vs dynamic (dynamic-exponent) 8-bit format…
Read article →Bottleneck Theory of Deep Learning
The information bottleneck theory of deep learning: layers as a Markov chain, the two mutual informations I(X;T) and I(T;Y), the information plane, th…
Read article →Byte-Pair Encoding Deep Dive
A math-first deep dive on the Byte-Pair Encoding merge algorithm: the base vocabulary, frequency counting over adjacent symbol pairs, the greedy argma…
Read article →Blockwise Parallel Transformer
Blockwise Parallel Transformer (BPT) from first principles: why the feedforward network becomes the memory bottleneck once FlashAttention removes the …
Read article →Capability Evaluation Overview
The math of LLM capability evaluations: accuracy as a Bernoulli estimate, standard error, Wald vs Wilson vs Clopper-Pearson confidence intervals, samp…
Read article →Capability Transitions in Training
The math of capability phase transitions: why some skills appear to switch on sharply with scale while the loss falls smoothly. Sigmoid vs step curves…
Read article →Causal Tracing
Causal tracing from first principles: the three-run recipe of clean, corrupted, and corrupted-with-restoration passes that Meng et al. used in ROME to…
Read article →Classifier-Free Guidance (CFG)
The math of classifier-free guidance: the score-function view of diffusion, how Bayes' rule turns a conditional and unconditional model into an i…
Read article →Chinchilla Scaling Law
The Chinchilla scaling law as a strategic turning point: how Kaplan-era advice built a generation of giant under-trained models, how Hoffmann et al. (…
Read article →Chinchilla Scaling
The Chinchilla compute-optimal scaling result (Hoffmann et al. 2022): the parametric loss law L(N,D) = E + A/N^α + B/D^β, minimizing it under the comp…
Read article →Chroma Math + Architecture
The math and architecture behind Chroma: the embedded, in-process design that makes it the SQLite of vector stores, how it delegates search to hnswlib…
Read article →Chunked Prefill
Chunked prefill in LLM serving: why one long prompt stalls every decode in the batch, how splitting the prompt into fixed-size chunks and piggybacking…
Read article →Circuit Analysis
The math of transformer circuits: the residual stream as a shared communication channel, attention heads as low-rank read/write maps, the factoring of…
Read article →Classifier Guidance Math
Classifier guidance for diffusion models, derived from first principles: the score-function view, Bayes' rule splitting the conditional score, th…
Read article →Coding Capability Evals
How code-generation ability is actually measured: HumanEval and MBPP, the pass@k metric and its unbiased estimator, unit-test execution and functional…
Read article →ColBERT Math
The math of ColBERT and late interaction: per-token embeddings for query and document instead of one pooled vector, the MaxSim scoring formula (max co…
Read article →Communication Math for Distributed Training
The collective communication primitives behind distributed training: all-reduce, all-gather, reduce-scatter, broadcast, and all-to-all; the ring all-r…
Read article →Compute-Bound Operations
The compute-bound regime in transformer inference and training: the FLOP ceiling, arithmetic intensity and the roofline ridge point, why prefill and l…
Read article →Compute-Optimal Training in Practice
A practitioner's recipe for compute-optimal training: how to read an isoFLOP frontier, allocate a fixed FLOP budget across model size and tokens,…
Read article →Compute-Optimal Training: the 6ND Formula and Chinchilla
Compute-optimal training from first principles: the C = 6ND FLOP formula, a Lagrangian solve for model size and tokens at fixed compute, and the Chinc…
Read article →Constitutional AI Math
How Constitutional AI (CAI) and RLAIF replace human harmlessness labels with AI feedback steered by a written constitution: the two phases (supervised…
Read article →Context Length Extension Math
The math behind extending a pretrained transformer's context window: why RoPE breaks out of distribution past its training length, Position Inter…
Read article →Contextual Retrieval Math: Contextual Embeddings and BM25
Contextual retrieval: prepend LLM-written context to each chunk, then contextual embeddings, contextual BM25 and reranking cut retrieval failures ~67%…
Read article →Continuous Batching Math
The math of continuous (iteration-level) batching for LLM serving: why static batching wastes GPU, how Orca-style per-iteration admit/evict keeps the …
Read article →Contrastive Search
Contrastive search decoding from first principles: the model-confidence minus degeneration-penalty objective, the alpha tradeoff, the max cosine simil…
Read article →Corrective RAG (CRAG)
Corrective RAG (CRAG): a lightweight retrieval evaluator scores the relevance of retrieved documents, two thresholds sort them into correct, ambiguous…
Read article →CPU Offload Math
The arithmetic of CPU offload for training and running transformers on one GPU: what lives in GPU memory (parameters, gradients, Adam optimizer state)…
Read article →Cross-Entropy Loss at Scale: Log-Sum-Exp, Fused, Vocab-Parallel
Cross-entropy loss in LLM training as a system: log-sum-exp in bf16, the fused p - y gradient, masked token reduction, vocab-parallel CE, z-loss, perp…
Read article →Cyber Capability Evals
How to measure the cyber-offense capability of a language model: CTF and vulnerability-discovery benchmarks, decomposing an attack into scorable sub-t…
Read article →Dangerous Capability Evaluations
The general framework for dangerous-capability evaluations and responsible scaling: threat modeling from harm scenario to measurable proxy, capability…
Read article →Training Data Filtering
A focused walk through pretraining data filtering: heuristic rules, classifier-based quality filters, perplexity filtering with an n-gram language mod…
Read article →Training Data Mixing
The math of pretraining data-mixture optimization: the mixture as a distribution over domains, token-proportional baselines, upsampling and downsampli…
Read article →Data Quality Math
How to put numbers on training-data quality: reference-model perplexity and classifier scores as quality metrics, the deduplication math behind effect…
Read article →Data Quality Scaling
How data quality changes scaling: curation and filtering shift the loss-vs-compute curve leftward and downward, so the same loss is reached with fewer…
Read article →Data Scaling for LLMs
Data-constrained scaling: what happens when you run out of unique tokens. The Muennighoff et al. result that repeating data for up to ~4 epochs is nea…
Read article →DDIM: Denoising Diffusion Implicit Models, Update Formula
DDIM explained: the non-Markovian forward process, the DDIM update formula, eta between DDIM and DDPM, deterministic sampling in fewer steps, inversio…
Read article →DDP Math
The math of Distributed Data Parallel training: why averaging gradients across N replicas is exactly the gradient of the loss on the union batch, grad…
Read article →DDPM Math: Forward Process, Variational Bound, Epsilon Loss
Denoising diffusion probabilistic models (DDPM) derived: forward noising, the closed-form marginal, posterior, variational bound, epsilon-prediction M…
Read article →Deception Capability Evals
How to actually measure deception in language models: the metrics behind sandbagging tests, sycophancy flip rates, strategic-deception scenarios, cons…
Read article →DeepMind Scaling Law Series
How DeepMind reshaped LLM scaling: the Gopher 280B baseline, the isoFLOP experimental method, and the three independent estimation approaches in Hoffm…
Read article →Dense Retrieval Math
The math of dense retrieval: the dual-encoder (bi-encoder) architecture mapping queries and documents into a shared embedding space, similarity as dot…
Read article →Depth Scaling in Transformers: More Layers vs More Width
Depth scaling in transformers: parameters grow linearly with layers, deep stacks are hard to train, pre-norm and DeepNet fix that, and depth vs width.
Read article →Dictionary Learning for NN
Dictionary learning for neural-network interpretability: the overcomplete generative model, the sparse-coding objective, the two-phase alternating opt…
Read article →DoRA
DoRA (Weight-Decomposed Low-Rank Adaptation), from the mechanics up: how it splits every weight matrix into a magnitude vector and a direction matrix,…
Read article →DoRA Theory
The theory behind DoRA (Weight-Decomposed Low-Rank Adaptation): why decoupling a weight's magnitude from its direction matters, the learning-dyna…
Read article →Double Descent
Double descent explained from first principles: the classic bias-variance U-curve, the interpolation threshold where test error peaks, and the second …
Read article →Double Descent Deep Dive
A mathematical deep dive on double descent: the bias-variance decomposition in the overparameterized regime, a solvable random-features linear-regress…
Read article →DPM-Solver Explained: Fast Diffusion Sampling in 10-20 Steps
How DPM-Solver samples diffusion models in 10-20 steps: the probability-flow ODE, log-SNR variables, exponential integrators, DDIM as order 1, DPM-Sol…
Read article →DPO Math
A deep dive on the Direct Preference Optimization loss mechanics: the log-sigmoid of the implicit-reward margin, the gradient and its self-modulating …
Read article →DyT
The math of Dynamic Tanh (DyT): the normalization-free transformer layer DyT(x) = gamma * tanh(alpha * x) + beta that replaces LayerNorm and RMSNorm. …
Read article →E5 Embedding Model Explained: The Contrastive Training Math
How the E5 embedding model is trained: CCPairs web pairs, consistency filtering, InfoNCE with in-batch negatives, temperature, query and passage prefi…
Read article →EAGLE
A deep dive into EAGLE speculative decoding beyond the base idea: the exact feature-autoregression recurrence, the one-layer draft head, EAGLE-2'…
Read article →EAGLE in Context
The math of EAGLE speculative decoding: drafting at the feature level (second-to-top hidden states) with a small autoregressive head that conditions o…
Read article →EDM Diffusion (Karras et al.): Preconditioning, Sampling
EDM diffusion by Karras et al. explained: the sigma parameterization, c_skip, c_out, c_in and c_noise preconditioning, the rho schedule, and the Heun …
Read article →Effective Batch Size
Effective (global) batch size = micro-batch x gradient-accumulation x data-parallel world. How the identity decouples the batch your optimizer sees fr…
Read article →Elasticsearch kNN Vector Search: HNSW, Quantization, Hybrid
How Elasticsearch kNN vector search works in Lucene: dense_vector, HNSW, num_candidates, filtered kNN, hybrid BM25 with RRF, int8 and BBQ quantization…
Read article →Token Embeddings in Transformers: Tokens to Vectors
How transformer token embeddings work: a lookup in the vocab x d_model matrix, embedding geometry, positional information, weight tying with the outpu…
Read article →Emergent Capabilities Deep Dive
A critical, quantitative look at emergent abilities in large language models: the claim that capabilities appear abruptly at scale, the mirage argumen…
Read article →Emergent Abilities in LLMs
A foundational overview of emergent abilities in large language models: the Wei et al. definition, the canonical examples (multi-digit arithmetic, wor…
Read article →Emergent Abilities Math
The mathematics of emergent abilities: how a smoothly improving per-token accuracy p turns into a sharp jump once you measure exact-match success as p…
Read article →EP Math
The systems math of expert parallelism for Mixture-of-Experts: the all-to-all dispatch and combine communication volume, how experts are placed across…
Read article →Expert Choice Routing
Expert-choice routing for Mixture-of-Experts: the transposed selection where experts pick their top-k tokens instead of tokens picking experts, the ga…
Read article →Expert Parallel Math Deep
A math deep dive on expert parallelism for Mixture-of-Experts models: routing as an expectation over a batch, why imbalance is a parallelism problem r…
Read article →Faiss Math
A practical guide to FAISS as an approximate-nearest-neighbor toolkit: the index zoo and how to choose. IndexFlat exact search, IVF inverted-file coar…
Read article →Feature Disentanglement
The math of feature disentanglement in interpretability: the linear representation hypothesis and features-as-directions, the interference term, super…
Read article →Transformer Math: Big Picture
A capstone map of the whole Transformer Math series: the residual stream and embeddings, attention and the transformer block, training math, scaling l…
Read article →Fine-Tuning Scaling
How fine-tuning scales: the effective-data-transfer law from Hernandez et al., how downstream performance depends on pretraining compute and fine-tune…
Read article →Flash Attention Architecture
A first-principles overview of FlashAttention (FA-1): why standard attention is memory-bound, the HBM-vs-SRAM memory hierarchy, IO-aware tiling, the o…
Read article →FlashAttention-2
FlashAttention-2 in depth: why FlashAttention-1 only reached a fraction of peak FLOPs, the 16x cost gap between matmul and non-matmul work, deferring …
Read article →Flash Attention Math
A from-first-principles derivation of the FlashAttention online-softmax recurrence: the running max m_i, the running denominator l_i, the rescaling id…
Read article →Flow Matching Explained: Velocity Fields, Rectified Flow, ODEs
Flow matching for generative models: probability paths, the velocity-field ODE, the conditional flow matching loss, straight-line rectified flow, vs d…
Read article →FP4 Quantization
Post-training 4-bit weight quantization for inference: NF4 (4-bit NormalFloat, quantile-optimal for Gaussian weights, from QLoRA) and FP4 (E2M1), why …
Read article →FP4 Training on Blackwell
FP4 training explained: the 4-bit E2M1 float and its sixteen representable values, the extreme range and precision limits of four bits, fine-grained m…
Read article →FP8 Training on H100+ Hardware
How FP8 training works: the E4M3 and E5M2 formats, their bit layouts and representable ranges, per-tensor and delayed (amax-history) scaling with a wo…
Read article →FSDP Math
How PyTorch FSDP implements ZeRO-3-style full sharding: the FlatParameter sharding unit and wrapping policy, the forward all-gather/compute/reshard an…
Read article →Full FT vs PEFT Trade-offs
Full fine-tuning versus parameter-efficient fine-tuning as a decision, worked in numbers: the 16-bytes-per-parameter memory bill of full FT, why freez…
Read article →Fusion RAG
RAG-Fusion explained from first principles: generating multiple sub-queries from one question, retrieving a ranked list for each, and merging those li…
Read article →Game Theory for LLM Agents
A focused, game-theoretic reading of LLM interactions: framing agents as players, Nash equilibrium and best response, negotiation and debate as games,…
Read article →GELU vs SiLU vs SwiGLU: Transformer Activation Functions
GELU vs SiLU vs SwiGLU: GELU as x times Phi(x), its erf and tanh forms, a worked example, why it replaced ReLU, and how gated SiLU and SwiGLU differ.
Read article →GGUF File Format: Layout, K-Quants, Q4_K, imatrix
What a GGUF file is: the llama.cpp single-file header and tensor layout, mmap loading, k-quant super-blocks with a worked Q4_K example, and the imatri…
Read article →Distillation Math
The mathematics of knowledge distillation: temperature-softened softmax, the KL-divergence objective, the combined soft/hard loss, the T-squared gradi…
Read article →GLU Variants in Transformer FFNs: SwiGLU, GEGLU, ReGLU Compared
The GLU feed-forward family in transformers: FFN(x) = (act(xW) * xV)W2 with GLU, ReGLU, GEGLU and SwiGLU, the 2/3 hidden-width rule, params, why gatin…
Read article →GPTQ Math
The math of GPTQ post-training quantization: the layer-wise output-error objective, the second-order Hessian (OBQ lineage) formulation, greedy per-col…
Read article →GQA Math
The math of grouped-query attention: sharing K/V projections across groups of query heads, the g-groups design that interpolates between multi-head (g…
Read article →Gradient Accumulation Math
The math of gradient accumulation: the accumulate-then-step loop, why you divide the loss by K, how gradient linearity makes K micro-batches equal one…
Read article →Gradient Checkpointing Math
The math of gradient checkpointing: why activation memory grows linearly with depth, how recomputation trades FLOPs for memory, the sqrt(L) optimal ch…
Read article →Gradient Clipping in LLM Training: Global Norm and AMP Order
How gradient clipping by global norm stops exploding gradients in LLM training: clip by norm vs by value, unscale-then-clip in mixed precision, clip f…
Read article →Gradient Memory Math
The gradient as a concrete mathematical object: its shape and norm, the gradient as a vector field over parameter space, how per-layer gradient scale …
Read article →Graph RAG Math
The math of GraphRAG: extracting entities and relations into a knowledge graph, community detection with the Leiden algorithm and modularity, hierarch…
Read article →Grokking in Neural Networks: Delayed Generalization Explained
Grokking explained: why a network fits mod-97 training data by step ~10^3 yet validates at chance until ~10^5, and how weight decay and data fraction …
Read article →Grokking Deep Dive
A mechanistic deep dive into grokking: how a one-layer transformer trained on modular addition reverse-engineers into a Fourier-multiplication circuit…
Read article →Grokking Math
A theoretical account of grokking: why weight decay drives delayed generalization, the norm-growth-versus-generalization tradeoff, the two competing s…
Read article →What Is Grouped-Query Attention (GQA)? GQA vs MQA
What is grouped-query attention (GQA)? Query heads share fewer K and V heads, shrinking the KV cache: choosing the group count, uptraining, GQA vs MQA…
Read article →GTE Embeddings Math
The math behind GTE (General Text Embeddings) from Alibaba: the bi-encoder with mean pooling, L2 normalization and cosine similarity, the improved con…
Read article →Guided Generation Math
The math of constrained / guided LLM decoding: turning a regex or JSON schema into a finite-state machine over the vocabulary, logit masking, grammar-…
Read article →HNSW Math
The math of HNSW (Hierarchical Navigable Small World) graphs: the multi-layer navigable-small-world structure, greedy search descending the layers, pr…
Read article →Hoffmann Chinchilla Paper
The 2024 Epoch AI replication of Hoffmann et al. (2022): reconstructing the parametric loss-fit data from Figure 4, why the published Approach 3 estim…
Read article →Hybrid Architectures
Hybrid attention + SSM architectures: why a handful of attention layers rescues a recurrent stack, the interleaving ratio, the FLOP and KV-cache arith…
Read article →Hybrid Parallelism Math
How to compose data, tensor, pipeline, and expert parallelism into one training run: the device mesh, factoring N = DP x TP x PP, the communication-vo…
Read article →Hybrid Retrieval Math
The math of hybrid retrieval: why dense (semantic) and sparse (lexical) retrieval are complementary, and how to fuse their ranked lists. Reciprocal Ra…
Read article →HyDE Math
The math of HyDE (Hypothetical Document Embeddings): why dense retrieval suffers a query-document embedding gap, how generating a hypothetical answer …
Read article →Hyena
The math of the long convolution, the primitive Hyena uses in place of attention. Causal convolution as a lower-triangular Toeplitz operator, why filt…
Read article →Hyena Hierarchy Math
The math of Hyena: replacing attention with a recurrence of long implicit convolutions and data-controlled gating. How the long filter is parameterize…
Read article →IA3 Theory
IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): the three learned scaling vectors on keys, values, and the FFN intermediate, the…
Read article →Inference-Optimal Training
Why the Chinchilla compute-optimal point ignores inference cost, and how minimizing total lifetime cost -- training FLOPs (6ND) plus expected inferenc…
Read article →Inference Scaling Laws: Test-Time Compute, Best-of-N, pass@N
How inference scaling laws work: accuracy vs test-time compute, the log-linear best-of-N curve, pass@N coverage, compute-optimal allocation, train vs …
Read article →In-Flight Batching
In-flight batching in TensorRT-LLM: the runtime's iteration-level scheduler, mixing prefill and decode requests in one batch via the attention pl…
Read article →Information Bottleneck Method: IB Lagrangian, Beta, Info Curve
The information bottleneck of Tishby et al.: min I(X;T) - beta I(T;Y), self-consistent equations, Blahut-Arimoto iteration, information curve, rate-di…
Read article →Weight Initialization Scaling: Xavier, Kaiming, 1/sqrt(2N), muP
How transformer weight init is scaled: variance preservation, the fan-in rule, Xavier/Glorot and Kaiming/He, GPT-2 0.02 std, 1/sqrt(2N) residuals, muP…
Read article →Mechanistic Interpretability Overview
A map of mechanistic interpretability: the ladder of methods from attribution and probing up through activation patching, circuit analysis, and sparse…
Read article →Iteration-Level Scheduling
Iteration-level scheduling in LLM serving: why the scheduling quantum is a single forward pass, the per-iteration admit/run/evict loop, FCFS versus pr…
Read article →Kaplan Scaling Laws: Loss vs N, D, Compute (6ND)
Kaplan et al. 2020 scaling laws: test loss as a power law in parameters, data and compute across seven orders of magnitude, the 6ND rule, and Chinchil…
Read article →Kaplan Scaling Laws: Loss vs Parameters, Data, Compute
Kaplan et al. 2020 scaling laws explained: power laws for loss vs parameters, data and compute, the N^0.73 allocation rule, batch size, why Chinchilla…
Read article →KTO Math
A math-first deep dive on KTO (Kahneman-Tversky Optimization): aligning a language model with a prospect-theory value function using binary desirable …
Read article →KV cache math architecture
A first-principles overview of the KV cache: why autoregressive decoding recomputes without it, the O(N^2) to O(N) saving, the bytes-per-token sizing …
Read article →KV Cache Deep Dive
KV cache internals in depth: the per-layer tensor shapes [batch, heads, seq, head_dim], memory-layout choices and strides, how the cache is indexed an…
Read article →KV Cache Compression
KV-cache compression beyond quantization: token eviction (H2O, Scissorhands), attention sinks and sliding windows (StreamingLLM), low-rank projection,…
Read article →KV Cache Quantization at Low Bits
A deep dive on aggressive KV-cache quantization: why keys and values need asymmetric treatment, per-channel key vs per-token value quantization, the o…
Read article →Label smoothing architecture
Deep-dive on label smoothing: why one-hot cross-entropy causes runaway logits and overconfidence, the 1-ε soft target and its gradient, the calibratio…
Read article →Layer-wise Learning Rate Decay (LLRD) for Fine-Tuning
Layer-wise learning rate decay (LLRD) for fine-tuning transformers: the per-layer rule, why lower layers get smaller steps, a 12-layer BERT example, g…
Read article →Layer Normalization in Transformers: Formula, Backprop
Layer normalization in a transformer: the per-token mean and variance formula, gamma and beta, a worked example, LayerNorm vs batch norm, the backward…
Read article →LayerNorm vs RMSNorm architecture
An implementation-focused look at LayerNorm and RMSNorm: epsilon placement, fp16/bf16 mixed precision, one-pass and Welford variance, catastrophic can…
Read article →Learning Rate Math for LLM
The math of the learning rate itself: its role as the step size in gradient descent, the loss-landscape and curvature view, the maximum stable step si…
Read article →Linear Attention
A guided tour of linear attention and its variants: the kernel feature-map factorization that turns O(N^2) into O(N), and how Linear Transformer, Perf…
Read article →Linear Attention Math
The math of linear attention: replacing softmax(QK^T)V with a kernel feature map phi so attention factorizes as phi(Q)(phi(K)^T V), the associativity …
Read article →Lion Optimizer Math
The math of Lion (EvoLved Sign Momentum): the sign-of-interpolated-momentum update rule, the two EMA coefficients, why it stores only one optimizer st…
Read article →LLM Inference Architecture: The Full Pipeline in Depth
An end-to-end map of the LLM serving stack: the request lifecycle from tokenize to prefill to the decode loop to detokenize, the engine components (sc…
Read article →LLM.int8(): Outlier Features and Mixed-Precision Decomposition
The LLM.int8() math: vector-wise absmax INT8, emergent outlier features past 6.7B parameters, and the fp16 mixed-precision decomposition, with worked …
Read article →Lookahead Decoding
Lookahead decoding as transformer math: the Jacobi-iteration view of autoregressive decoding, the evolving n-gram pool, a lookahead branch that refine…
Read article →LoRA
The mathematics of Low-Rank Adaptation: the update W' = W + (alpha/r)BA, the low-rank factorization, rank r and scaling alpha, the trainable para…
Read article →LoRA Theory
The theory behind Low-Rank Adaptation: the intrinsic-dimensionality hypothesis (Aghajanyan et al.), why fine-tuning weight updates live in a low-rank …
Read article →LLM Training Loss Curve: How to Read Warmup, Spikes, Plateaus
How to read an LLM training loss curve: power-law descent on log-log axes, the warmup bump, noise floor, loss spikes, plateaus and the train-validatio…
Read article →Loss Scaling for fp16 Training
Loss scaling for FP16 training: why FP16 gradients underflow to zero in the subnormal range, the FP16 range math, and the scale-then-unscale algorithm…
Read article →Lottery Ticket Hypothesis
A foundational overview of the Lottery Ticket Hypothesis: Frankle and Carbin's claim that a dense network contains a sparse subnetwork which, tra…
Read article →Lottery Ticket Hypothesis
A deep dive into the modern Lottery Ticket Hypothesis: iterative magnitude pruning with rewinding to iteration k, linear mode connectivity and the ins…
Read article →Lottery Ticket Hypothesis
The formal side of the lottery ticket hypothesis: the strong LTH, the Malach et al. proof that a randomly initialized network already contains an appr…
Read article →Learning Rate Schedules for LLM
A focused guide to learning-rate schedule shapes for training transformers: why warmup exists, linear vs cosine decay, why cosine must be matched to t…
Read article →Learning Rate Transfer
How to tune the learning rate on a small proxy model and transfer it to a large one: standard parametrization drifts the optimal LR with width, the ma…
Read article →Mamba SSM Math
The math of Mamba's selective state-space model: the SSM recurrence h_t = A h_(t-1) + B x_t and y_t = C h_t, the selectivity innovation (input-de…
Read article →Math Capability Evals
How mathematical reasoning is measured in language models: the benchmark ladder from GSM8K to MATH to AIME and olympiad problems, answer-matching vers…
Read article →Matmul Arithmetic Intensity
The arithmetic intensity of matrix multiply from first principles: FLOPs/byte = 2MNK / bytes moved, how M, N, and K set data reuse, the roofline ridge…
Read article →Matryoshka Embeddings
How Matryoshka Representation Learning (MRL) trains embeddings so nested prefixes — the first 64, 128, 256, ... dimensions — are each independently us…
Read article →Medusa Speculative Heads
A deep dive on Medusa speculative decoding: the extra heads as residual blocks, the conditional-independence gap that makes them decay with distance, …
Read article →Medusa
The math of Medusa decoding: bolting multiple lightweight residual heads onto a frozen model so each predicts a future token, expanding their top-k ou…
Read article →LLM Memory Math
How to compute the memory a language model needs to train: the four buckets (parameters, gradients, optimizer state, activations), why mixed-precision…
Read article →Mesa-Optimization
Mesa-optimization explained: the distinction between the base optimizer that trains a model and a learned optimizer that can emerge inside it, mesa-ob…
Read article →Micro-Batch Size Selection
Micro-batch size as the unit of work that must fit in memory: how it drives peak activation memory, how it trades against GEMM/kernel efficiency, and …
Read article →Milvus Math + Architecture
The math and architecture behind Milvus: the disaggregated, log-first design that separates compute from storage, collections and partitions and segme…
Read article →Min-P Sampling
Min-p sampling explained from first principles: the relative threshold p_base * max_prob, why scaling the cutoff with the model's confidence prun…
Read article →Mirostat Sampling: Target Surprise tau and the mu Update
Mirostat sampling: a feedback controller that holds surprise at tau, a Zipf fit for the exponent s, solving for top-k, the mu update rule, and Mirosta…
Read article →Mixed-precision training math
An overview of mixed-precision transformer training: fp16/bf16 compute copies over an fp32 master-weight spine, why gradient updates round to zero in …
Read article →MoE math architecture
The core math of Mixture-of-Experts layers: the router softmax, top-K selection, the sparse FFN forward pass, active vs total parameters, sparse-vs-de…
Read article →MQA Math
Multi-Query Attention (MQA): all query heads share one key/value head, cutting the KV cache by a factor of h. The shapes and math, the KV-cache reduct…
Read article →Multi-Agent Communication
The token-cost math of communication between LLM agents: message passing, the context-budget tradeoff of shared versus private state, star / chain / f…
Read article →Multi-Agent Coordination
How multiple LLM agents coordinate: decomposing a goal into a task graph, allocating tasks to agents, orchestrator/worker versus decentralized pattern…
Read article →Multi-Agent Debate
Multi-agent debate as a truthfulness and accuracy mechanism: several model instances argue, then a judge or a vote decides. Why debate can amplify cor…
Read article →Multi-Agent Emergence
Emergent collective behavior in multi-agent LLM systems: the analogy to swarms and complex adaptive systems, how division of labor and roles arise wit…
Read article →Multi-Agent Negotiation
Negotiation mechanics for LLM agents: utility functions and reservation values, the bargaining zone (ZOPA), offer and counter-offer protocols, Pareto-…
Read article →Multi-Agent Planning
How a team of LLM agents plans together: joint versus individual plans, the hierarchical planner-executor split, decomposing a goal into subgoals, a s…
Read article →Multi-Agent Reflection
Reflection in multi-agent systems: self-critique and cross-agent critique loops, the Reflexion pattern of verbal reinforcement, actor-evaluator-self-r…
Read article →Multi-Agent Shared Memory
Shared memory in multi-agent LLM systems: the blackboard / shared-state architecture versus message passing, consistency and staleness, memory as shar…
Read article →Multi-Agent System Theory
A unifying theory of multi-agent LLM systems: when several agents beat one, the decomposition-versus-overhead tradeoff, a taxonomy tying communication…
Read article →Multi-Hop RAG
Multi-hop retrieval-augmented generation: why one retrieval fails on compositional questions, query decomposition into sub-questions, the iterative re…
Read article →Multi-Head Attention Explained: Why Split Heads Beat One Head
Multi-head attention: split d_model into h heads, attend in each subspace, concatenate and project with W_O. Why it adds no parameters and beats one w…
Read article →Multilingual Scaling
How languages scale together inside one model: the curse of multilinguality and capacity dilution, positive cross-lingual transfer versus interference…
Read article →Multilingual Tokenization
Why the same sentence costs 2-4x more tokens in some languages than in English: tokenizer fertility and tokens-per-word disparity across scripts, the …
Read article →Multimodal Scaling
Multimodal scaling laws: per-modality power laws, competition and synergy between modalities under a shared parameter and compute budget, the data-mix…
Read article →Muon optimizer architecture
Deep-dive on the Muon optimizer: momentum, orthogonalizing 2D weight updates toward U V^T, the quintic Newton-Schulz iteration, RMS-to-RMS shape scali…
Read article →muP
muP as a parametrization: the per-layer initialization-variance and learning-rate scaling rules that keep every activation, logit, and gradient update…
Read article →Neural Scaling Law Theory
Why neural scaling laws are power laws, not just that they are: the data-manifold-dimension theory (Sharma & Kaplan, alpha = 4/d where d is the in…
Read article →Neural Tangent Kernel
An intuitive overview of the Neural Tangent Kernel: why very wide neural networks behave almost linearly in their parameters, how training turns into …
Read article →Neural Tangent Kernel (NTK): Infinite-Width Training Math
The neural tangent kernel (NTK) defined, why it stays constant at infinite width, training as kernel gradient descent, the closed form, GPs and spectr…
Read article →NTK-Aware Interpolation
NTK-aware RoPE scaling explained from first principles: why rotary frequencies break outside the training window, the spectral-bias argument for leavi…
Read article →NVMe Offload Math
The arithmetic of NVMe/SSD offload for training giant transformers: how ZeRO-Infinity spills parameters and optimizer state to solid-state disk, the c…
Read article →Offload Math
The general framework for offloading model state off the GPU: the GPU-HBM to CPU-DRAM to NVMe memory-tier hierarchy, the overlap-vs-stall condition t_…
Read article →OpenAI Scaling Law Contributions
OpenAI's broader scaling program beyond the Kaplan power laws: GPT-2 zero-shot task transfer, the compute-efficient frontier, why larger models a…
Read article →OpenSearch Vector Search
The math behind OpenSearch vector search and the k-NN plugin: the knn_vector field, the three engines (Lucene, nmslib, Faiss) and when each wins, spac…
Read article →Optimized Product Quantization
Optimized Product Quantization (OPQ): why PQ's fixed subspace split is suboptimal when variance is unevenly spread across dimensions, how OPQ lea…
Read article →Optimizer State Offload Math
The math of optimizer-state offload: why Adam's momentum and variance are the biggest, coldest bucket of training memory, how ZeRO-Offload pushes…
Read article →Adam Optimizer Memory: Bytes per Parameter in Training
How much memory Adam optimizer state takes: bytes per parameter for SGD, momentum and Adam, the fp32 master copy, a 7B worked example, 8-bit Adam and …
Read article →Optimizer Math for LLM Training
A unifying overview of the optimizers used to train transformers: the general update template, the SGD to momentum to RMSProp to Adam progression, ada…
Read article →PagedAttention architecture
Deep-dive on PagedAttention, the KV-cache management scheme behind vLLM that treats the cache like OS virtual memory. Covers why KV-cache fragmentatio…
Read article →Paged Attention Deep Dive
A deep dive on PagedAttention: how vLLM stores the KV cache in fixed-size blocks with a per-sequence block table, eliminates internal and external fra…
Read article →PagedAttention
Why naive contiguous KV-cache allocation wastes 60-80% of GPU memory to internal and external fragmentation and max-length over-reservation, and how P…
Read article →Paged KV Cache Deep Dive
A deep dive on the paged KV-cache block manager: choosing block size (tokens per block) and its fragmentation-vs-overhead trade-off, the block table a…
Read article →PEFT Theory Overview
The unifying theory behind parameter-efficient fine-tuning: why full fine-tuning is over-parameterized, the intrinsic-dimension hypothesis, the additi…
Read article →Persuasion Capability Evals
How to measure a language model's persuasion capability: the estimand of attitude and belief shift, randomized human-subject designs, control ver…
Read article →pgvector Math + Architecture
The math and architecture of pgvector, the Postgres vector extension: the vector type, the L2 / cosine / inner-product operator classes, exact versus …
Read article →Pinecone Math + Architecture
A focused look at Pinecone, the managed vector database: serverless vs pod-based architecture, namespaces, the black-box managed ANN index, metadata f…
Read article →Pipeline Parallel Math: Deriving the GPipe Bubble Step by Step
Pipeline parallelism math: layers split into stages, the naive bubble, GPipe microbatching, deriving (p-1)/(m+p-1), a worked example, 1F1B schedule.
Read article →Polysemanticity
Polysemanticity: why a single neuron in a transformer responds to many unrelated features, how it differs from its cause (superposition), how to measu…
Read article →Position Interpolation Math
Linear Position Interpolation (PI) for RoPE, derived from first principles: why direct extrapolation puts rotation angles out of distribution, the m -…
Read article →Pipeline Parallelism Bubble (p-1)/(m+p-1): GPipe vs 1F1B
Derive the pipeline parallelism bubble fraction (p-1)/(m+p-1), see how micro-batches shrink it, and compare GPipe, 1F1B and interleaved schedules on m…
Read article →PPO for LLMs Math
How PPO drives RLHF for LLMs: framing generation as a token-level RL problem, the reward-model-minus-KL-penalty reward, GAE advantage estimation, the …
Read article →PQ Math
The math of Product Quantization (PQ): splitting a vector into m sub-vectors, learning a k-means codebook per subspace, encoding a vector as m codewor…
Read article →Prefill Math
The math of the prefill phase: processing the whole prompt in one parallel forward pass to populate the KV cache and emit the first token, why prefill…
Read article →Prefix Tuning Explained: Per-Layer Prefixes vs Prompt Tuning
Prefix tuning: trainable key and value prefixes at every transformer layer, the reparameterization MLP, parameter counts, a GPT-2 example, vs prompt t…
Read article →Pre-norm vs post-norm transformers
Deep-dive on transformer normalization placement: post-norm on the residual path versus pre-norm inside the branch, gradient flow and attenuation, war…
Read article →Pretraining Scaling
How pretraining loss falls with compute, the C = 6ND accounting identity, planning a pretraining run from a fixed FLOP budget, token budgets and data …
Read article →Priority Scheduling
Priority scheduling for LLM serving: priority classes and strict priority queues, weighted fair queuing, preemption of low-priority requests and KV-ca…
Read article →Linear Probes
Probing classifiers for interpretability: linear and MLP probes trained on frozen activations to test what information is linearly decodable, why prob…
Read article →Prompt Prefill vs Decode
The decode phase in depth: why generating one token at a time is memory-bandwidth-bound with arithmetic intensity near 1, the per-token latency formul…
Read article →Prompt Lookup Decoding Math
The math of prompt lookup decoding (PLD): a zero-model speculative method that drafts from the prompt itself. How n-gram matching against the context …
Read article →Prompt Tuning Theory
Prompt tuning from first principles: relaxing discrete prompt search into a continuous soft-prompt matrix P: [p, d], where the gradient actually flows…
Read article →Qdrant Math + Architecture
The math behind Qdrant: collections, shards and segments; cosine as normalize-then-dot; the HNSW m / ef_construct / hnsw_ef budget and its memory bill…
Read article →QLoRA Theory
The theory behind QLoRA: why NF4 is a quantile grid rather than a uniform one, how double quantization recovers ~0.37 bits per parameter, the differen…
Read article →Quantization Quality Evaluation
How to evaluate a quantized LLM: perplexity delta and why it hides behavior changes, downstream task accuracy, per-layer error norms (MSE), signal-to-…
Read article →Advanced RAG Techniques Overview
The math of advanced RAG: query rewriting and multi-query expansion, why sparse and dense scores cannot be added directly, reciprocal rank fusion work…
Read article →Reasoning Capability Evals
The statistics of reasoning benchmarks: why GPQA Diamond, MMLU-Pro, ARC-AGI and BBH are structurally small, chance-corrected accuracy and the guessing…
Read article →Reasoning Model Scaling
How reasoning accuracy scales with chain-of-thought length and sampling: self-consistency (majority vote over K sampled chains) and its accuracy-vs-K …
Read article →Rectified Flow Math: Straight-Line ODE Paths and Reflow
Rectified flow from first principles: linear noise-to-data interpolation, the velocity regression loss, marginal-preserving ODE, reflow for few-step s…
Read article →Redis Vector Search Math
The math behind Redis vector search and RediSearch: inverted-index posting-list memory, TF-IDF and BM25 scoring with a worked example, why cosine, inn…
Read article →Representation Learning Theory
The math of learned representations: the InfoNCE objective and its log N mutual-information bound, temperature, the alignment-uniformity decomposition…
Read article →Reranking Math
The retrieve-then-rerank pipeline explained from the math up: dense bi-encoder first-stage retrieval, the cross-encoder that jointly encodes query and…
Read article →RetNet
Retentive Networks (RetNet) explained from first principles: retention as a fading-memory replacement for softmax attention, the recurrent form with a…
Read article →RetNet Math
The math of RetNet (Retentive Network): retention as a decay-weighted alternative to softmax attention, its three equivalent forms (parallel for train…
Read article →Retrieval Architectures Overview
Retrieval fundamentals from first principles: the bi-encoder factorization that turns ranking into vector search, embedding geometry, why dot product,…
Read article →Reward Hacking Math
The math of reward over-optimization in RLHF: Goodhart's law, the KL-regularized objective and its Gibbs-tilted optimum, the empirical KL-vs-true…
Read article →Reward Model Math
How a reward model is trained for RLHF: bolting a scalar value head onto a pretrained language model, the Bradley-Terry preference model, the pairwise…
Read article →Reward Model Scaling
Reward normalization and scaling in RLHF: why raw reward-model scores need whitening before the PPO update, running mean/std normalization, per-batch …
Read article →Ring Attention
Ring Attention as a distributed systems problem: the activation and KV-cache memory bill at million-token context, where context parallelism sits amon…
Read article →Ring Attention Math Deep
Ring Attention explained from the math up: sharding a long sequence across P devices, keeping each query block fixed while K/V blocks rotate around a …
Read article →RLHF Math Deep Dive
A deep look at the RLHF objective itself: the KL-regularized reward-maximization objective max E[r(x,y)] - beta*KL(pi || pi_ref), why the KL leash is …
Read article →RLHF Scaling Laws: Reward Model Overoptimization and KL Budget
How RLHF scales with policy size, reward-model size and preference data: the reward model overoptimization law, proxy vs gold reward, the optimal KL b…
Read article →RLHF Scaling Math
The quantitative side of scaling RLHF: why sqrt(KL) is the natural coordinate for the reward-vs-drift frontier (a second-order Fisher argument), the b…
Read article →Scaling Laws for Reward Model Overoptimization: Goodhart in RLHF and Best-of-n
Measure and fit reward model overoptimization: the gold-versus-proxy setup, best-of-n and RL functional forms in sqrt(KL), peak formulas, an unbiased …
Read article →RMSNorm architecture - root-mean-square normalization in modern transformers
Deep-dive on RMSNorm: the root-mean-square formula, contrast with LayerNorm's mean-centering and bias, forward and backward passes, pre-norm resi…
Read article →RMSNorm Deep Dive
A deep dive on RMSNorm below the formula: the Jacobian as a rank d-1 projector, why the input gradient is exactly orthogonal to the input, a worked ba…
Read article →Roofline Model for LLM Ops
The roofline model applied to LLM inference: building the compute and bandwidth ceilings for a real machine, computing the ridge point, plotting a pre…
Read article →Rotary Position Embedding (RoPE)
The mechanism of Rotary Position Embedding, derived from scratch: the 2-D rotation-matrix and complex-exponential formulations, a proof that the inner…
Read article →Rotary Position Embedding Architecture in Depth
Positional encoding as a design problem: what a scheme must provide, how learned absolute, sinusoidal, T5 relative bias, ALiBi and rotary compare, whe…
Read article →RWKV Math: WKV Attention as an RNN, RWKV-5, -6 and -7
How the RWKV model works: token shift, the WKV operator as decay-weighted attention collapsing into a constant-size RNN state, RWKV-5/6/7 (Eagle, Finc…
Read article →Sparse Autoencoders (SAE)
Sparse autoencoders for transformer interpretability: the superposition hypothesis and why features outnumber dimensions, the encoder/decoder architec…
Read article →LLM Sampling Math: Temperature, Top-k, Top-p and Min-p Explained
The math of LLM sampling: temperature as log-odds scaling, why entropy rises with it, top-k vs top-p vs min-p truncation, and repetition penalties.
Read article →Scaled Dot-Product Attention: Why Divide by sqrt(d_k)
The scaled dot-product attention formula softmax(QK^T/sqrt(d_k))V: the variance argument for 1/sqrt(d_k), softmax saturation, a worked example, maskin…
Read article →Scaling law architecture
How to use scaling laws as a design tool: the compute-optimal rule of thumb (C = 6ND, ~20 tokens per parameter), why compute-optimal is the wrong targ…
Read article →Scaling Laws Explained: Kaplan vs Chinchilla, Compute-Optimal
The math of neural scaling laws: why loss is a power law, the L(N, D) joint form and its floor, why Kaplan and Chinchilla differ, and the compute-opti…
Read article →Scaling Laws
Where neural scaling laws break down: data-constrained scaling and the decaying value of repeated tokens, the effective-data model, mixture-of-experts…
Read article →Hyperparameter Scaling Transfer
Hyperparameter transfer across model scale from first principles: why the optimal learning rate drifts with width under standard parametrization, the …
Read article →ScaNN
How ScaNN accelerates maximum inner-product search with anisotropic, score-aware vector quantization: why the residual component parallel to a datapoi…
Read article →Score Matching
Score matching from first principles: why the normalizing constant makes maximum likelihood intractable, what the score function ∇_x log p(x) is, Hyva…
Read article →Search Scaling in Reasoning
Verifier-guided search as a test-time scaling axis: Best-of-N with a reward model and the coverage ceiling 1-(1-p)^N, why flat sampling wastes compute…
Read article →Selective Activation Recomputation: Which Tensors to Recompute
Selective recomputation math: rank tensors by recompute FLOPs per byte freed, why norms and attention scores are cheap to rebuild, 5% vs 33% slowdown.
Read article →Self-Consistency Scaling
The math of self-consistency: majority voting over k independently sampled chains of thought. Derives the vote-accuracy curve, shows why the real bar …
Read article →Self-RAG
Self-RAG from first principles: reflection tokens as a learned vocabulary, the retrieve-or-not decision and when retrieval actively hurts, per-passage…
Read article →SentencePiece Tokenizer Explained: Unigram, BPE, Byte Fallback
SentencePiece tokenization in depth: raw text to ids, the whitespace meta-symbol, NFKC normalization, byte fallback, unigram vs BPE, and vocabulary si…
Read article →Sequence Length Math
The cost math of sequence length in a transformer: the quadratic attention term versus the linear projection and FFN term, the crossover at s = 6d, th…
Read article →Sequence Parallel Math
Sequence parallelism explained with the math: partitioning activations along the sequence dimension to cut activation memory, the Megatron-style SP th…
Read article →Shampoo
How Shampoo works: the full-matrix AdaGrad ideal and why it is intractable, Kronecker factorization of the preconditioner into left and right factors,…
Read article →Shared KV Cache
Cross-layer KV sharing: cutting the KV cache by reusing keys and values across transformer LAYERS, not just across heads. How Cross-Layer Attention (C…
Read article →Signal Propagation in Deep Networks
Signal propagation theory for deep networks: the variance recursion through a layer and its fixed point, the mean-field order-chaos phase transition a…
Read article →Sliding Window Attention Math: Mistral, Rolling KV Cache, O(N*w)
Sliding window attention math (Mistral-style): O(N*w) instead of O(N^2), a rolling-buffer KV cache capped at w tokens, and a receptive field growing w…
Read article →SLO-Aware Scheduling Math
The queueing and scheduling math behind LLM latency SLOs: TTFT versus TPOT, prefill/decode interference, sizing a chunked prefill, utilization versus …
Read article →SmoothQuant Math
The math of SmoothQuant: enabling INT8 weight-and-activation (W8A8) quantization by migrating quantization difficulty from activations to weights with…
Read article →SOAP Optimizer: Running Adam in Shampoo's Eigenbasis
How SOAP works: why Shampoo with power 1/2 equals Adafactor in its eigenbasis, the reference algorithm step by step, numpy code, a toy comparison with…
Read article →Softmax numerics architecture
How softmax is actually computed: the overflow range of fp16 and bf16, the max-subtraction identity and its proof, why the sum is accumulated in fp32,…
Read article →Sophia Optimizer
How Sophia builds a cheap diagonal curvature preconditioner for language-model pre-training: the clipped second-order update rule, the Hutchinson and …
Read article →SP Math
Sequence parallelism as a configuration decision: what the SP knob actually buys, why its degree is welded to the tensor-parallel degree, how it compa…
Read article →Sparse Retrieval Math
Sparse retrieval from first principles: the bag-of-words vector and the inverted index, TF-IDF and the full BM25 formula with term-frequency saturatio…
Read article →Speculative Decoding in Practice
A systems deep dive on speculative decoding: choosing a draft model and the tokenizer-mismatch trap, self-speculation with Medusa heads and EAGLE feat…
Read article →Speculative Decoding Math: Acceptance Rate and Draft Length
The math of speculative decoding: the min(1, p/q) rejection rule, proof the output matches the target, acceptance rate alpha, speedup, draft length.
Read article →SPLADE Math
The math of SPLADE learned sparse retrieval: projecting hidden states through the masked-language-model head into vocabulary space, the log(1 + ReLU(w…
Read article →State Space Models Math
The math of linear time-invariant state space models (SSMs): the continuous system x' = Ax + Bu, y = Cx + Du, zero-order-hold and bilinear discre…
Read article →Selective SSM
How selective state space models work: making Delta, B and C functions of the input so the recurrence becomes time-varying, per-token ZOH discretizati…
Read article →Stable Diffusion Math
How Stable Diffusion works as a system: latent diffusion and the compute arithmetic that justifies it, the VAE encoder/decoder and the 0.18215 scale f…
Read article →State Space Models
The state space model (SSM) foundation behind S4, S4D and Mamba: the continuous linear system x'(t) = Ax(t) + Bu(t), ZOH and bilinear discretizat…
Read article →Stochastic Rounding
Stochastic rounding explained for low-precision training: round up with probability equal to frac(x/ulp) so the rounded value is unbiased, E[round(x)]…
Read article →Non-Autoregressive Structured Outputs: Parallel Sampling vs Constrained Decoding and JSON Mode, and Why a Typed Decision Model Cannot Emit a Type Error
The probability behind structured LLM output: rejection and parallel sampling as the exact valid-output conditional, why per-step constrained decoding…
Read article →Superposition Hypothesis: Polysemantic Neurons and Toy Models
The superposition hypothesis explained: how networks pack more sparse features than dimensions, near-orthogonal directions, interference, polysemantic…
Read article →SwiGLU Math: Gated FFN Formula and a Llama-7B Worked Example
SwiGLU math: the SiLU-gated FFN in Llama and PaLM, (Swish(xW) * xV)W2, why it needs three matrices, the 2/3 d_ff rule and a Llama-7B parameter count.
Read article →Switch Transformer Math: Top-1 Routing, Capacity, Balance Loss
Switch Transformer math: top-1 expert routing, the capacity factor formula and token dropping, the load-balancing loss, all-to-all dispatch, fp32 rout…
Read article →Synthetic Training Data
The mathematics of training on model-generated data: the recursive model-collapse recursion and a worked variance-shrinkage example, tail loss from fi…
Read article →Temperature Sampling Math
How sampling temperature acts on generation quality, measured rather than asserted: the asymmetric convergence rates at the cold and hot limits, the e…
Read article →LLM Temperature Math: Dividing Logits by T Before Softmax
What LLM temperature does mathematically: logits divided by T before softmax, low vs high temperature, T=0 as greedy decoding, top-k and top-p order, …
Read article →Tensor Parallel Math Deep
A first-principles derivation of Megatron-style tensor parallelism: splitting a single matmul across GPUs with column-parallel then row-parallel linea…
Read article →Tensor Programs (Yang’s Framework)
Tensor Programs explained: what a tensor program is, what the master theorem says about infinite-width limits, the abc-parametrization of init scale, …
Read article →Test-Time Compute Math
Test-time (inference-time) compute scaling: spending more compute at inference to get better answers. The four ways to spend it -- longer chains of th…
Read article →How tiktoken Works: Byte-Level BPE, cl100k vs o200k
What tiktoken does under the hood: byte-level BPE with 256 base bytes, the pre-tokenization regex, the Rust merge loop, special tokens, and cl100k vs …
Read article →Top-K Sampling Math
The math of top-k sampling: the fixed-cardinality truncation rule and its renormalization, the k-th logit as an order statistic, why a fixed k imposes…
Read article →Top-p (Nucleus) Sampling Math: Why It Beats a Fixed Top-k
Top-p vs top-k from first principles: the cumulative-mass cutoff, why retained mass overshoots p, and why an adaptive nucleus size beats a fixed k per…
Read article →Transfer Learning Scaling
Transfer and downstream scaling laws: how pretraining scale turns into fine-tuning and downstream performance, the effective-data-transferred power la…
Read article →Typical Sampling
Locally typical sampling derived from first principles: the typical set and the asymptotic equipartition property, the uniform-information-density arg…
Read article →Unigram LM Tokenizer
The unigram language model tokenizer as a probabilistic model: the generative story of a sentence as independently drawn subword pieces, marginalizing…
Read article →Variance Maintenance in Deep Nets
The practical engineering of variance maintenance in deep transformers: what to instrument (per-layer activation RMS, the residual-stream growth curve…
Read article →Vector DB Math
How a vector database actually works: embedding storage and its memory cost, the ANN index (HNSW / IVF / PQ as pointers), distance metrics (cosine, do…
Read article →Verifier Scaling
Verifier-guided and best-of-N scaling: the accuracy-vs-N curve, why a good verifier beats majority vote, outcome vs process reward models, verifier-gu…
Read article →Vespa Math
The math that makes Vespa different from a vector store: first-class tensor fields and tensor ranking expressions, the multi-phase ranking cost equati…
Read article →LR Warmup Deep Dive
Why the first few hundred steps of transformer training need a smaller step size: Adam's thin second-moment estimate and what bias correction doe…
Read article →Weaviate Math + Architecture
The arithmetic behind Weaviate: what a collection stores per object, named vectors, the dynamic-ef formula, thresholds measured in objects rather than…
Read article →Width Scaling
How transformer quality and training dynamics change as you widen d_model at fixed depth: the quadratic parameter law (about 12 * L * d^2 non-embeddin…
Read article →YaRN + NTK Scaling
YaRN explained from first principles: RoPE frequencies and wavelengths, why linear position interpolation crushes the high-frequency dimensions, the N…
Read article →ZeRO Stage 1 Math
How ZeRO Stage 1 partitions Adam optimizer states across data-parallel ranks: the 16-bytes-per-parameter memory model, why the fp32 master weights, mo…
Read article →ZeRO Stage 2 Math
ZeRO Stage 2 explained with the memory math: partition gradients as well as optimizer states so each rank keeps only its gradient shard, cutting gradi…
Read article →ZeRO Stage 3 Math
The memory and communication math of ZeRO Stage 3: full parameter partitioning that puts 1/N of every parameter, gradient, and optimizer state on each…
Read article →Zero-Bubble Pipeline Parallelism: Bubble Fraction, B/W Split
Pipeline bubble fraction in GPipe and 1F1B, and how zero-bubble scheduling splits backward into input-gradient B and weight-gradient W to fill it: ZB-…
Read article →ZeRO-Infinity
ZeRO-Infinity as a synthesis: the arithmetic-intensity threshold that decides which memory tier can hold which state, bandwidth-centric partitioning a…
Read article →ZeRO Math
ZeRO (Zero Redundancy Optimizer) explained from the memory model up: the redundancy in vanilla data parallelism where every GPU holds full parameters,…
Read article →ZeRO-Offload Explained: Moving Adam Optimizer State to the CPU
ZeRO-Offload math: moving fp32 Adam optimizer states and the update to CPU, the 16-bytes-per-parameter budget, PCIe cost, and a 10B model on one GPU.
Read article →Token Embedding Lookup: How the Embedding Matrix Works
How a token embedding lookup works: the vocab x d_model matrix, why a row lookup equals one-hot times E, parameter count, sparse gradients and weight …
Read article →Tokenizer Math
A first-principles deep-dive on tokenizer math: why subword tokenization sits between characters and words, the frequency-based BPE merge algorithm wi…
Read article →Training Data for Small Language Models: Chinchilla, Dedup
Picking SLM training data: the Chinchilla 20 tokens per parameter rule vs over-training, 6ND token math, MinHash dedup, quality filters, mixing, synth…
Read article →Weight Initialization
Why weight initialization matters: variance preservation through layers to avoid vanishing and exploding activations and gradients, the forward and ba…
Read article →How Weights Are Stored on Disk
How model weights are stored and laid out in memory: row-major vs column-major order, strides and address arithmetic, why layout decides cache localit…
Read article →