All 383 articles, sorted alphabetically
The Autoregressive Generation Loop
How a language model actually produces text: the autoregressive factorization p(sequence) = prod p(x_t | x_<t), the two phases of infer…
Read article →Backpropagation Through Transformers
How backpropagation and the chain rule train transformers: forward vs backward pass, the chain rule for composed functions, the vector-Jacobian-produc…
Read article →Beam Search
The mathematics of beam search decoding: approximate search for the highest-probability sequence, accumulating log-probabilities along beams, beam wid…
Read article →The Complete Transformer Block
How a complete pre-norm transformer block composes attention, a feed-forward network, two residual connections, and two normalizations end to end: the…
Read article →CPU Cache Hierarchy and Transformer Inference
How the CPU memory hierarchy -- registers, L1, L2, L3, and DRAM -- decides small-language-model inference speed on a CPU. Latency and bandwidth for ea…
Read article →CPU Inference Pipelines
The end-to-end CPU serving path for LLMs and small language models: tokenize, prefill, the autoregressive decode loop, and detokenize. Why prefill is …
Read article →CPU Matmul Kernels and BLAS
How a fast CPU matmul (GEMM) is really built: why GEMM is the transformer workhorse, the naive triple loop and its cache-miss problem, loop reordering…
Read article →CPU Memory Budget for SLM Training
A full accounting of the CPU RAM budget for running a small language model: model weights (params x bytes by precision), the KV cache (2 x L x n_kv x …
Read article →Cross-Entropy Loss for Next-Token Prediction
Cross-entropy loss for language modeling from first principles: maximum likelihood and negative log-likelihood, the one-hot target and -log p(correct …
Read article →Dataloader and Tokenization Pipeline
How the data loading and tokenization pipeline feeds a language model: raw text to tokens, document packing with separators into fixed-length sequence…
Read article →The Geometry of Dot Products
The geometry of the dot product and why attention is built on it: the dot product as projection and as similarity, the a·b = |a||b|…
Read article →End-to-End CPU SLM Recipe
A single worked recipe that ties the whole series together: build and run a ~100M-parameter small language model destined for CPU inference. Pick an a…
Read article →Feed-Forward Network
A first-principles deep-dive on the position-wise feed-forward network (FFN/MLP) in a transformer block: the two linear layers with a 4x expansion fac…
Read article →Fine-Tuning Math
The math of fine-tuning: the memory ledger of full fine-tuning (weights + gradients + optimizer states = ~16 bytes/param for Adam in fp32), why that c…
Read article →FlashAttention
A first-principles walk through FlashAttention: the IO-aware, tiled attention algorithm, online (streaming) softmax with a running max and running nor…
Read article →Gradient Accumulation and Microbatches
How gradient accumulation simulates a large effective batch on limited memory by summing gradients over K microbatches before an optimizer step: the e…
Read article →Gradient Clipping and Training Stability
Gradient clipping and training stability from first principles: why gradients explode and loss spikes appear, global-norm clipping (scale g by min(1, …
Read article →KV Cache
The exact math of the transformer KV cache: why autoregressive decode would recompute every past key and value without it, what the cache stores, the …
Read article →KV Cache Quantization
A deep dive on KV cache quantization: why the stored keys and values dominate long-context decode memory, the bytes-per-token math and how it scales w…
Read article →LayerNorm
The math of layer normalization: normalizing across the feature dimension per token, the mean and variance, (x - mu)/sqrt(sigma^2 + epsilon), the lear…
Read article →Learning Rate Schedules
A first-principles deep dive on learning-rate schedules for training transformers: why a constant learning rate fails, linear warmup and why it stabil…
Read article →Linear Algebra for Transformers
The linear algebra that underlies transformers, from first principles: vectors and matrices, matrix multiplication as the workhorse operation, dot pro…
Read article →Long Context Strategies
A taxonomy of long-context strategies with the complexity math for each: the two costs that explode with sequence length -- O(N^2) attention compute a…
Read article →Diagnosing Loss Curves
A practical guide to reading training and validation loss curves: the healthy power-law decay shape, learning-rate-too-high divergence and NaN spikes,…
Read article →Mixed Precision Training
A first-principles deep dive on mixed-precision training: the FP32, FP16, and BF16 bit layouts, the exponent-vs-mantissa (range-vs-precision) trade, w…
Read article →Mixture of Experts
The math of Mixture-of-Experts: replacing a dense FFN with N experts and a softmax router, top-k sparse activation, the decoupling of total parameters…
Read article →Multi-Head Attention
The math of multi-head attention: why multiple heads give the model different representation subspaces, splitting d_model into h heads of size d_k = d…
Read article →Multi-Token Prediction (MTP)
A deep dive on multi-token prediction (MTP): predicting several future tokens per position instead of one, multiple prediction heads sharing a trunk, …
Read article →Output Projection and Logits
How a transformer turns its final hidden state into next-token probabilities: the output projection (unembedding / LM head) mapping d_model to vocab, …
Read article →Perplexity and Evaluation Metrics
A first-principles deep dive on perplexity and language-model evaluation: perplexity as exp(cross-entropy), the branching-factor intuition, a fully wo…
Read article →Positional Encoding
The math of positional encoding: why self-attention is permutation-equivariant and order-blind, the original sinusoidal encoding and its 10000^(2i/d) …
Read article →Quantization Layouts in Memory
How quantized transformer weights are actually laid out in memory: the affine quantization map q = round(x/scale) + zero_point, symmetric vs asymmetri…
Read article →Residual Connections and Gradient Flow
The math of residual connections in transformers: the y = x + F(x) skip, why it creates a gradient highway (dL/dx = dL/dy (1 + F')), …
Read article →RLHF and DPO
The math of RLHF and DPO: preference data and the Bradley-Terry model, learning a reward model, the KL-constrained reward-maximization objective and i…
Read article →RMSNorm
The math of RMSNorm: normalizing a vector by its root-mean-square only, with no mean subtraction and no bias — y_i = x_i / sqrt(mean…
Read article →Extending RoPE to Longer Contexts
How to extend a RoPE model's context window: a brief recap of rotary position embedding (rotating Q/K by position-dependent angles), …
Read article →Sampling Strategies
The math of sampling from a language model: how logits become a probability distribution via softmax, greedy/argmax decoding, temperature scaling and …
Read article →Scaled Dot-Product Attention
Scaled dot-product attention from first principles: how Q, K, and V are projected from the input, the QK^T score matrix, why we divide by sqrt(d_k) (t…
Read article →SGD vs Adam vs AdamW Optimizer Math
A first-principles walk through optimizer math: plain SGD, momentum as an exponential moving average of gradients, RMSProp as an EMA of squared gradie…
Read article →SIMD Instructions for Transformer Math
How SIMD vectorization speeds up transformer kernels on the CPU: one instruction over many lanes, AVX2/AVX-512 (8/16 fp32 lanes) and ARM NEON, how the…
Read article →SLM Architectures
A deep dive comparing modern small language model architectures across the design axes that actually differentiate them: depth vs width, attention var…
Read article →Softmax Derivation
Deriving softmax from first principles: turning a vector of logits into a positive normalized probability distribution, why the exponential is the rig…
Read article →Speculative Decoding
The math of speculative decoding: a small draft model proposes k tokens, the large target verifies them in one parallel forward pass, and modified rej…
Read article →The Future of CPU SLM in 2026 and Beyond
Why CPU-based small language models are becoming genuinely viable: sub-4-bit and QAT quantization, stronger small models from over-training and distil…
Read article →Tied Embeddings and Parameter Sharing
How weight tying shares the input embedding matrix with the output projection: the parameter savings (one vocab x d_model matrix instead of two), the …
Read article →3D Parallelism
How data, tensor, and pipeline parallelism combine to train giant models: the N = DP x TP x PP factorization of the GPU count, the standard placement …
Read article →Ablation Study Math
The math behind ablation studies in LLM research: treatment effects, compute- and parameter-matched controls, seed variance and standard error, confid…
Read article →Beyond ReLU
GeLU, SiLU, and SwiGLU compared from first principles: exact formulas and derivatives, why smooth and gated activations beat ReLU in transformers, the…
Read article →Activation checkpointing architecture
Deep-dive on activation (gradient) checkpointing: why activations dominate training memory, how the forward pass saves only segment boundaries and dro…
Read article →Activation Memory Math
How to count activation memory in a transformer from first principles: which tensors autograd must save and why, the per-layer 34sbh + 5as^2b accounti…
Read article →Activation Patching
Activation patching from first principles: the clean/corrupted two-prompt setup, the intervention written as math, logit-difference and normalized-rec…
Read article →Activation Quantization
Why activations are harder to quantize than weights: per-input dynamic range, outlier channels and tokens, static vs dynamic quantization, per-tensor …
Read article →Activation Recomputation Math
The math of activation recomputation in transformer training: the 34sbh + 5as^2b activation-memory formula, the one-third FLOPs overhead of full recom…
Read article →Activation Scaling
How activation variance propagates through a transformer: the fan-in variance identity, Xavier and He init, why the residual stream accumulates varian…
Read article →Adafactor
Adafactor from first principles: why Adam's second moment costs a full extra copy of the model, how a rank-1 factorization under the …
Read article →Adam and AdamW -- the optimizer that trains transformers
Deep-dive on Adam and AdamW: the single-learning-rate limitation, momentum (first moment, smoothed gradient), adaptive per-parameter scaling (second m…
Read article →AdamW Math Deep Dive
The math of AdamW: why L2 regularization in the gradient and weight decay on the weights are the same thing under SGD but provably different under Ada…
Read article →Adapter Theory
The math of bottleneck adapters for parameter-efficient fine-tuning: the down-project / nonlinearity / up-project structure, near-identity initializat…
Read article →Adaptive RAG
Adaptive RAG explained from first principles: deciding whether to retrieve at all, routing queries by predicted complexity into no-retrieval, single-s…
Read article →Admission Control Math
The math of admission control for LLM serving: Little's law, M/M/1 latency blowup near saturation, why token-length variance makes LL…
Read article →Agent Scaling Laws
How LLM agent capability scales: the geometric horizon model S(H) = p^H, why per-step error compounds, the H_50 task-horizon metric and its doubling t…
Read article →ALiBi
How ALiBi (Attention with Linear Biases) encodes position by adding a linear, head-specific distance penalty to the raw attention scores instead of us…
Read article →Alignment Tax Math
The alignment tax: the capability-benchmark regression that can accompany RLHF, why it happens (KL drift from the pretrained distribution, reward-mode…
Read article →Annoy
How Annoy (Approximate Nearest Neighbors Oh Yeah) works: random-projection binary trees that split space by the perpendicular bisector of two sampled …
Read article →MongoDB Atlas Vector Search
The math behind MongoDB Atlas Vector Search: why exact kNN is O(N*d), how the HNSW graph reaches sublinear search, what M, efConstruction and numCandi…
Read article →Attention compute and memory -- the quadratic cost of context
Deep-dive on attention compute and memory: the O(N^2) QK^T attention matrix, quadratic compute and memory, the attention-vs-FFN FLOPs breakdown, compu…
Read article →Attention Math
Attention derived from first principles: start with a hard dictionary lookup, relax it into a differentiable soft lookup, and the query-key-value form…
Read article →Attention variants architecture
A comparative map of attention variants: multi-head, multi-query, grouped-query, multi-head latent, sliding-window, sparse and linear attention, score…
Read article →AWQ Math
The math of AWQ (Activation-aware Weight Quantization): why a small set of salient weight channels identified by activation magnitude dominate quality…
Read article →Backpressure Math
The math of backpressure in an LLM serving pipeline: queue depth as the integral of arrival-minus-service rate, Little's law applied …
Read article →Backprop architecture
The systems architecture of the transformer backward pass: what the chain rule actually costs in memory and bandwidth, gradient accumulation as a batc…
Read article →Bandwidth-Bound Operations
Why autoregressive decoding is memory-bandwidth-bound, not compute-bound: the arithmetic intensity of GEMV, bytes-per-token accounting for weights and…
Read article →Batched GEMM Math
Batched GEMM in transformer workloads: the shape algebra of per-head attention matmuls, strided-batched stride arithmetic, grouped GEMM for MoE and ra…
Read article →Batch Size Math for LLM Training
The math of batch size in LLM training: the gradient as a noisy estimator and its 1/B variance, the gradient noise scale and critical batch size, the …
Read article →Beam Search Math
The math of beam search decoding: MAP inference over sequences, the log-prob scoring recurrence, length normalization derived, the beam-search curse, …
Read article →BF16 Training
BF16 training in depth: the bfloat16 bit layout (1 sign, 8 exponent, 7 mantissa), why matching FP32's exponent range makes it a drop-…
Read article →BGE Embeddings Math
The math behind BGE and BGE-M3 text embeddings: the bi-encoder architecture, CLS versus mean pooling, L2 normalization and cosine similarity, contrast…
Read article →Bio Capability Evals
How to measure the biology and bio-risk capabilities of large language models: the difference between knowledge probes and task-based evals, the uplif…
Read article →BitFit Theory
BitFit explained from first principles: freeze every weight matrix and train only the bias vectors. Which biases are updated, the exact parameter coun…
Read article →bitsandbytes int8 Quantization
How bitsandbytes does 8-bit: block-wise absmax quantization and why blocks bound outlier damage, the linear vs dynamic (dynamic-exponent) 8-bit format…
Read article →Bottleneck Theory of Deep Learning
The information bottleneck theory of deep learning: layers as a Markov chain, the two mutual informations I(X;T) and I(T;Y), the information plane, th…
Read article →Byte-Pair Encoding Deep Dive
A math-first deep dive on the Byte-Pair Encoding merge algorithm: the base vocabulary, frequency counting over adjacent symbol pairs, the greedy argma…
Read article →Blockwise Parallel Transformer
Blockwise Parallel Transformer (BPT) from first principles: why the feedforward network becomes the memory bottleneck once FlashAttention removes the …
Read article →Capability Evaluation Overview
The math of LLM capability evaluations: accuracy as a Bernoulli estimate, standard error, Wald vs Wilson vs Clopper-Pearson confidence intervals, samp…
Read article →Capability Transitions in Training
The math of capability phase transitions: why some skills appear to switch on sharply with scale while the loss falls smoothly. Sigmoid vs step curves…
Read article →Causal Tracing
Causal tracing from first principles: the three-run recipe of clean, corrupted, and corrupted-with-restoration passes that Meng et al. used in ROME to…
Read article →Classifier-Free Guidance (CFG)
The math of classifier-free guidance: the score-function view of diffusion, how Bayes' rule turns a conditional and unconditional mod…
Read article →Chinchilla Scaling Law
The Chinchilla scaling law as a strategic turning point: how Kaplan-era advice built a generation of giant under-trained models, how Hoffmann et al. (…
Read article →Chinchilla Scaling
The Chinchilla compute-optimal scaling result (Hoffmann et al. 2022): the parametric loss law L(N,D) = E + A/N^α + B/D^&…
Read article →Chroma Math + Architecture
The math and architecture behind Chroma: the embedded, in-process design that makes it the SQLite of vector stores, how it delegates search to hnswlib…
Read article →Chunked Prefill
Chunked prefill in LLM serving: why one long prompt stalls every decode in the batch, how splitting the prompt into fixed-size chunks and piggybacking…
Read article →Circuit Analysis
The math of transformer circuits: the residual stream as a shared communication channel, attention heads as low-rank read/write maps, the factoring of…
Read article →Classifier Guidance Math
Classifier guidance for diffusion models, derived from first principles: the score-function view, Bayes' rule splitting the condition…
Read article →Coding Capability Evals
How code-generation ability is actually measured: HumanEval and MBPP, the pass@k metric and its unbiased estimator, unit-test execution and functional…
Read article →ColBERT Math
The math of ColBERT and late interaction: per-token embeddings for query and document instead of one pooled vector, the MaxSim scoring formula (max co…
Read article →Communication Math for Distributed Training
The collective communication primitives behind distributed training: all-reduce, all-gather, reduce-scatter, broadcast, and all-to-all; the ring all-r…
Read article →Compute-Bound Operations
The compute-bound regime in transformer inference and training: the FLOP ceiling, arithmetic intensity and the roofline ridge point, why prefill and l…
Read article →Compute-Optimal Training in Practice
A practitioner's recipe for compute-optimal training: how to read an isoFLOP frontier, allocate a fixed FLOP budget across model size…
Read article →Compute-Optimal Training
The compute-optimal frontier from first principles: the C = 6ND compute identity (2 forward + 4 backward FLOPs per parameter per token), the Lagrangia…
Read article →Constitutional AI Math
How Constitutional AI (CAI) and RLAIF replace human harmlessness labels with AI feedback steered by a written constitution: the two phases (supervised…
Read article →Context Length Extension Math
The math behind extending a pretrained transformer's context window: why RoPE breaks out of distribution past its training length, Po…
Read article →Contextual RAG Math
The math of contextual retrieval: why chunks lose their meaning, how an LLM prepends document-level context, contextual embeddings and contextual BM25…
Read article →Continuous Batching Math
The math of continuous (iteration-level) batching for LLM serving: why static batching wastes GPU, how Orca-style per-iteration admit/evict keeps the …
Read article →Contrastive Search
Contrastive search decoding from first principles: the model-confidence minus degeneration-penalty objective, the alpha tradeoff, the max cosine simil…
Read article →Corrective RAG (CRAG)
Corrective RAG (CRAG): a lightweight retrieval evaluator scores the relevance of retrieved documents, two thresholds sort them into correct, ambiguous…
Read article →CPU Offload Math
The arithmetic of CPU offload for training and running transformers on one GPU: what lives in GPU memory (parameters, gradients, Adam optimizer state)…
Read article →Cross-entropy loss architecture
Cross-entropy for language models as an engineering system, not a formula: the fused softmax-CE kernel and the p - y gradient, why the logits tensor d…
Read article →Cyber Capability Evals
How to measure the cyber-offense capability of a language model: CTF and vulnerability-discovery benchmarks, decomposing an attack into scorable sub-t…
Read article →Dangerous Capability Evaluations
The general framework for dangerous-capability evaluations and responsible scaling: threat modeling from harm scenario to measurable proxy, capability…
Read article →Training Data Filtering
A focused walk through pretraining data filtering: heuristic rules, classifier-based quality filters, perplexity filtering with an n-gram language mod…
Read article →Training Data Mixing
The math of pretraining data-mixture optimization: the mixture as a distribution over domains, token-proportional baselines, upsampling and downsampli…
Read article →Data Quality Math
How to put numbers on training-data quality: reference-model perplexity and classifier scores as quality metrics, the deduplication math behind effect…
Read article →Data Quality Scaling
How data quality changes scaling: curation and filtering shift the loss-vs-compute curve leftward and downward, so the same loss is reached with fewer…
Read article →Data Scaling for LLMs
Data-constrained scaling: what happens when you run out of unique tokens. The Muennighoff et al. result that repeating data for up to ~4 epochs is nea…
Read article →DDIM
The math behind DDIM: a non-Markovian forward process that shares DDPM's marginals, deterministic sampling, the probability-flow ODE …
Read article →DDP Math
The math of Distributed Data Parallel training: why averaging gradients across N replicas is exactly the gradient of the loss on the union batch, grad…
Read article →DDPM
The math of DDPMs from first principles: the forward noising process q, the closed-form marginal, the tractable posterior, the reverse process p, the …
Read article →Deception Capability Evals
How to actually measure deception in language models: the metrics behind sandbagging tests, sycophancy flip rates, strategic-deception scenarios, cons…
Read article →DeepMind Scaling Law Series
How DeepMind reshaped LLM scaling: the Gopher 280B baseline, the isoFLOP experimental method, and the three independent estimation approaches in Hoffm…
Read article →Dense Retrieval Math
The math of dense retrieval: the dual-encoder (bi-encoder) architecture mapping queries and documents into a shared embedding space, similarity as dot…
Read article →Depth Scaling
Depth scaling in transformers: how adding layers (holding width fixed) changes capability and trainability, why parameter count is linear in depth, th…
Read article →Dictionary Learning for NN
Dictionary learning for neural-network interpretability: the overcomplete generative model, the sparse-coding objective, the two-phase alternating opt…
Read article →DoRA
DoRA (Weight-Decomposed Low-Rank Adaptation), from the mechanics up: how it splits every weight matrix into a magnitude vector and a direction matrix,…
Read article →DoRA Theory
The theory behind DoRA (Weight-Decomposed Low-Rank Adaptation): why decoupling a weight's magnitude from its direction matters, the l…
Read article →Double Descent
Double descent explained from first principles: the classic bias-variance U-curve, the interpolation threshold where test error peaks, and the second …
Read article →Double Descent Deep Dive
A mathematical deep dive on double descent: the bias-variance decomposition in the overparameterized regime, a solvable random-features linear-regress…
Read article →DPM-Solver Sampling
DPM-Solver explained from first principles: the semi-linear structure of the diffusion probability-flow ODE, the change of variables to log-SNR, the e…
Read article →DPO Math
A deep dive on the Direct Preference Optimization loss mechanics: the log-sigmoid of the implicit-reward margin, the gradient and its self-modulating …
Read article →DyT
The math of Dynamic Tanh (DyT): the normalization-free transformer layer DyT(x) = gamma * tanh(alpha * x) + beta that replaces LayerNorm and RMSNorm. …
Read article →E5 Embeddings Math
The math behind E5 text embeddings: weakly-supervised contrastive pre-training on the CCPairs corpus, consistency-based filtering, mean pooling, the q…
Read article →EAGLE
A deep dive into EAGLE speculative decoding beyond the base idea: the exact feature-autoregression recurrence, the one-layer draft head, EAGLE-2&a…
Read article →EAGLE in Context
The math of EAGLE speculative decoding: drafting at the feature level (second-to-top hidden states) with a small autoregressive head that conditions o…
Read article →EDM
EDM (Karras et al.) recast diffusion models around a single continuous noise level sigma: a unified sigma-parameterization, the c_skip / c_out / c_in …
Read article →Effective Batch Size
Effective (global) batch size = micro-batch x gradient-accumulation x data-parallel world. How the identity decouples the batch your optimizer sees fr…
Read article →Elasticsearch Vector Search
The math behind Elasticsearch vector search as Lucene implements it: the dense_vector field, per-segment HNSW graphs and why segment merging rebuilds …
Read article →Embeddings -- turning tokens into vectors
Deep-dive on transformer embeddings: token embeddings as a lookup into the embedding matrix (vocab x d_model), learned representations and embedding-s…
Read article →Emergent Capabilities Deep Dive
A critical, quantitative look at emergent abilities in large language models: the claim that capabilities appear abruptly at scale, the mirage argumen…
Read article →Emergent Abilities in LLMs
A foundational overview of emergent abilities in large language models: the Wei et al. definition, the canonical examples (multi-digit arithmetic, wor…
Read article →Emergent Abilities Math
The mathematics of emergent abilities: how a smoothly improving per-token accuracy p turns into a sharp jump once you measure exact-match success as p…
Read article →EP Math
The systems math of expert parallelism for Mixture-of-Experts: the all-to-all dispatch and combine communication volume, how experts are placed across…
Read article →Expert Choice Routing
Expert-choice routing for Mixture-of-Experts: the transposed selection where experts pick their top-k tokens instead of tokens picking experts, the ga…
Read article →Expert Parallel Math Deep
A math deep dive on expert parallelism for Mixture-of-Experts models: routing as an expectation over a batch, why imbalance is a parallelism problem r…
Read article →Faiss Math
A practical guide to FAISS as an approximate-nearest-neighbor toolkit: the index zoo and how to choose. IndexFlat exact search, IVF inverted-file coar…
Read article →Feature Disentanglement
The math of feature disentanglement in interpretability: the linear representation hypothesis and features-as-directions, the interference term, super…
Read article →Transformer Math: Big Picture
A capstone map of the whole Transformer Math series: the residual stream and embeddings, attention and the transformer block, training math, scaling l…
Read article →Fine-Tuning Scaling
How fine-tuning scales: the effective-data-transfer law from Hernandez et al., how downstream performance depends on pretraining compute and fine-tune…
Read article →Flash Attention Architecture
A first-principles overview of FlashAttention (FA-1): why standard attention is memory-bound, the HBM-vs-SRAM memory hierarchy, IO-aware tiling, the o…
Read article →FlashAttention-2
FlashAttention-2 in depth: why FlashAttention-1 only reached a fraction of peak FLOPs, the 16x cost gap between matmul and non-matmul work, deferring …
Read article →Flash Attention Math
A from-first-principles derivation of the FlashAttention online-softmax recurrence: the running max m_i, the running denominator l_i, the rescaling id…
Read article →Flow Matching
Flow matching for generative models: the probability path and velocity-field ODE, the continuity equation, the conditional flow matching objective, st…
Read article →FP4 Quantization
Post-training 4-bit weight quantization for inference: NF4 (4-bit NormalFloat, quantile-optimal for Gaussian weights, from QLoRA) and FP4 (E2M1), why …
Read article →FP4 Training on Blackwell
FP4 training explained: the 4-bit E2M1 float and its sixteen representable values, the extreme range and precision limits of four bits, fine-grained m…
Read article →FP8 Training on H100+ Hardware
How FP8 training works: the E4M3 and E5M2 formats, their bit layouts and representable ranges, per-tensor and delayed (amax-history) scaling with a wo…
Read article →FSDP Math
How PyTorch FSDP implements ZeRO-3-style full sharding: the FlatParameter sharding unit and wrapping policy, the forward all-gather/compute/reshard an…
Read article →Full FT vs PEFT Trade-offs
Full fine-tuning versus parameter-efficient fine-tuning as a decision, worked in numbers: the 16-bytes-per-parameter memory bill of full FT, why freez…
Read article →Fusion RAG
RAG-Fusion explained from first principles: generating multiple sub-queries from one question, retrieving a ranked list for each, and merging those li…
Read article →Game Theory for LLM Agents
A focused, game-theoretic reading of LLM interactions: framing agents as players, Nash equilibrium and best response, negotiation and debate as games,…
Read article →Activation function architecture
A focused deep-dive on the GELU activation: GELU(x) = x*Phi(x), the probabilistic stochastic-regularizer interpretation, the exact erf form and the ta…
Read article →GGUF Format Deep Dive
How the GGUF container and llama.cpp k-quants actually work: the single-file metadata-plus-tensors layout that makes GGUF mmap-friendly, block quantiz…
Read article →Distillation Math
The mathematics of knowledge distillation: temperature-softened softmax, the KL-divergence objective, the combined soft/hard loss, the T-squared gradi…
Read article →GLU FFN
The GLU FFN family: the gating template FFN(x) = (activation(xW) ⊙ xV)W2 and its members -- GLU (sigmoid), ReGLU (ReLU), GEGLU (G…
Read article →GPTQ Math
The math of GPTQ post-training quantization: the layer-wise output-error objective, the second-order Hessian (OBQ lineage) formulation, greedy per-col…
Read article →GQA Math
The math of grouped-query attention: sharing K/V projections across groups of query heads, the g-groups design that interpolates between multi-head (g…
Read article →Gradient Accumulation Math
The math of gradient accumulation: the accumulate-then-step loop, why you divide the loss by K, how gradient linearity makes K micro-batches equal one…
Read article →Gradient Checkpointing Math
The math of gradient checkpointing: why activation memory grows linearly with depth, how recomputation trades FLOPs for memory, the sqrt(L) optimal ch…
Read article →Gradient clipping architecture
Deep-dive on gradient clipping: why deep and recurrent networks explode gradients and NaN out runs, the clip-by-global-norm algorithm and how it prese…
Read article →Gradient Memory Math
The gradient as a concrete mathematical object: its shape and norm, the gradient as a vector field over parameter space, how per-layer gradient scale …
Read article →Graph RAG Math
The math of GraphRAG: extracting entities and relations into a knowledge graph, community detection with the Leiden algorithm and modularity, hierarch…
Read article →Grokking
A foundational overview of grokking: the delayed-generalization phenomenon Power et al. found on modular-arithmetic tasks, the long train/validation g…
Read article →Grokking Deep Dive
A mechanistic deep dive into grokking: how a one-layer transformer trained on modular addition reverse-engineers into a Fourier-multiplication circuit…
Read article →Grokking Math
A theoretical account of grokking: why weight decay drives delayed generalization, the norm-growth-versus-generalization tradeoff, the two competing s…
Read article →Grouped Query Attention (GQA)
A practitioner's guide to grouped-query attention: why cutting the number of K/V heads is the right efficiency lever, the arithmetic-…
Read article →GTE Embeddings Math
The math behind GTE (General Text Embeddings) from Alibaba: the bi-encoder with mean pooling, L2 normalization and cosine similarity, the improved con…
Read article →Guided Generation Math
The math of constrained / guided LLM decoding: turning a regex or JSON schema into a finite-state machine over the vocabulary, logit masking, grammar-…
Read article →HNSW Math
The math of HNSW (Hierarchical Navigable Small World) graphs: the multi-layer navigable-small-world structure, greedy search descending the layers, pr…
Read article →Hoffmann Chinchilla Paper
The 2024 Epoch AI replication of Hoffmann et al. (2022): reconstructing the parametric loss-fit data from Figure 4, why the published Approach 3 estim…
Read article →Hybrid Architectures
Hybrid attention + SSM architectures: why a handful of attention layers rescues a recurrent stack, the interleaving ratio, the FLOP and KV-cache arith…
Read article →Hybrid Parallelism Math
How to compose data, tensor, pipeline, and expert parallelism into one training run: the device mesh, factoring N = DP x TP x PP, the communication-vo…
Read article →Hybrid Retrieval Math
The math of hybrid retrieval: why dense (semantic) and sparse (lexical) retrieval are complementary, and how to fuse their ranked lists. Reciprocal Ra…
Read article →HyDE Math
The math of HyDE (Hypothetical Document Embeddings): why dense retrieval suffers a query-document embedding gap, how generating a hypothetical answer …
Read article →Hyena
The math of the long convolution, the primitive Hyena uses in place of attention. Causal convolution as a lower-triangular Toeplitz operator, why filt…
Read article →Hyena Hierarchy Math
The math of Hyena: replacing attention with a recurrence of long implicit convolutions and data-controlled gating. How the long filter is parameterize…
Read article →IA3 Theory
IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): the three learned scaling vectors on keys, values, and the FFN intermediate, the…
Read article →Inference-Optimal Training
Why the Chinchilla compute-optimal point ignores inference cost, and how minimizing total lifetime cost -- training FLOPs (6ND) plus expected inferenc…
Read article →Inference-Time Scaling Laws
Inference-time scaling laws: how accuracy grows with test-time compute, the log-linear best-of-N curve, the geometric coverage of pass@N, compute-opti…
Read article →In-Flight Batching
In-flight batching in TensorRT-LLM: the runtime's iteration-level scheduler, mixing prefill and decode requests in one batch via the …
Read article →Information Bottleneck Method
The foundational information bottleneck method of Tishby, Pereira and Bialek: the IB Lagrangian min I(X;T) - beta I(T;Y), the Markov constraint, the t…
Read article →Initialization Scaling
How weight-initialization variance is scaled in transformers: the variance-preservation principle, the fan-in rule, Xavier/Glorot and Kaiming/He init,…
Read article →Mechanistic Interpretability Overview
A map of mechanistic interpretability: the ladder of methods from attribution and probing up through activation patching, circuit analysis, and sparse…
Read article →Iteration-Level Scheduling
Iteration-level scheduling in LLM serving: why the scheduling quantum is a single forward pass, the per-iteration admit/run/evict loop, FCFS versus pr…
Read article →Kaplan Scaling Laws (2020)
The original Kaplan et al. (2020) neural scaling laws: the power-law relations of test loss to model size N, dataset size D, and compute C, the L = (N…
Read article →Kaplan Scaling Laws Deep Dive
A deep dive on Kaplan et al. 2020: the power-law fits for loss versus parameters, data, and compute; the N^0.73 compute-optimal allocation; the critic…
Read article →KTO Math
A math-first deep dive on KTO (Kahneman-Tversky Optimization): aligning a language model with a prospect-theory value function using binary desirable …
Read article →KV cache math architecture
A first-principles overview of the KV cache: why autoregressive decoding recomputes without it, the O(N^2) to O(N) saving, the bytes-per-token sizing …
Read article →KV Cache Deep Dive
KV cache internals in depth: the per-layer tensor shapes [batch, heads, seq, head_dim], memory-layout choices and strides, how the cache is indexed an…
Read article →KV Cache Compression
KV-cache compression beyond quantization: token eviction (H2O, Scissorhands), attention sinks and sliding windows (StreamingLLM), low-rank projection,…
Read article →KV Cache Quantization
A foundational overview of KV cache quantization: why the cache dominates long-context memory, the bytes-per-token math, int8 and int4 quantization of…
Read article →KV Cache Quantization at Low Bits
A deep dive on aggressive KV-cache quantization: why keys and values need asymmetric treatment, per-channel key vs per-token value quantization, the o…
Read article →Label smoothing architecture
Deep-dive on label smoothing: why one-hot cross-entropy causes runaway logits and overconfidence, the 1-ε soft target and its gradient, the calibratio…
Read article →Layer-wise LR Decay
Layer-wise learning-rate decay (LLRD) for fine-tuning transformers: the per-layer rule lr_l = base * ξ^(L-l), why lower layers dese…
Read article →Layer Normalization
Layer normalization from first principles: the per-token mean/variance formula, the learnable scale and shift, a worked numeric example, the backward-…
Read article →LayerNorm vs RMSNorm architecture
An implementation-focused look at LayerNorm and RMSNorm: epsilon placement, fp16/bf16 mixed precision, one-pass and Welford variance, catastrophic can…
Read article →Learning Rate Math for LLM
The math of the learning rate itself: its role as the step size in gradient descent, the loss-landscape and curvature view, the maximum stable step si…
Read article →Learning-rate schedules
Deep-dive on learning-rate schedules for transformer training: why warmup stabilizes adaptive optimizers, cosine vs inverse-sqrt vs linear decay, how …
Read article →Linear Attention
A guided tour of linear attention and its variants: the kernel feature-map factorization that turns O(N^2) into O(N), and how Linear Transformer, Perf…
Read article →Linear Attention Math
The math of linear attention: replacing softmax(QK^T)V with a kernel feature map phi so attention factorizes as phi(Q)(phi(K)^T V), the associativity …
Read article →Lion Optimizer Math
The math of Lion (EvoLved Sign Momentum): the sign-of-interpolated-momentum update rule, the two EMA coefficients, why it stores only one optimizer st…
Read article →LLM Inference Architecture: The Full Pipeline in Depth
An end-to-end map of the LLM serving stack: the request lifecycle from tokenize to prefill to the decode loop to detokenize, the engine components (sc…
Read article →LLM.int8() Paper Math
How LLM.int8() runs transformer matmuls in 8-bit at inference time: vector-wise (row/column) absmax INT8 quantization, the emergent-outlier-features p…
Read article →Lookahead Decoding
Lookahead decoding as transformer math: the Jacobi-iteration view of autoregressive decoding, the evolving n-gram pool, a lookahead branch that refine…
Read article →LoRA
The mathematics of Low-Rank Adaptation: the update W' = W + (alpha/r)BA, the low-rank factorization, rank r and scaling alpha, the tr…
Read article →LoRA Theory
The theory behind Low-Rank Adaptation: the intrinsic-dimensionality hypothesis (Aghajanyan et al.), why fine-tuning weight updates live in a low-rank …
Read article →Loss Curves
How to read and interpret an LLM training loss curve: the power-law descent in log-log axes, the warmup bump, the noise floor, spikes and divergence, …
Read article →Loss Scaling for fp16 Training
Loss scaling for FP16 training: why FP16 gradients underflow to zero in the subnormal range, the FP16 range math, and the scale-then-unscale algorithm…
Read article →Lottery Ticket Hypothesis
A foundational overview of the Lottery Ticket Hypothesis: Frankle and Carbin's claim that a dense network contains a sparse subnetwor…
Read article →Lottery Ticket Hypothesis
A deep dive into the modern Lottery Ticket Hypothesis: iterative magnitude pruning with rewinding to iteration k, linear mode connectivity and the ins…
Read article →Lottery Ticket Hypothesis
The formal side of the lottery ticket hypothesis: the strong LTH, the Malach et al. proof that a randomly initialized network already contains an appr…
Read article →Learning Rate Schedules for LLM
A focused guide to learning-rate schedule shapes for training transformers: why warmup exists, linear vs cosine decay, why cosine must be matched to t…
Read article →Learning Rate Transfer
How to tune the learning rate on a small proxy model and transfer it to a large one: standard parametrization drifts the optimal LR with width, the ma…
Read article →Mamba SSM Math
The math of Mamba's selective state-space model: the SSM recurrence h_t = A h_(t-1) + B x_t and y_t = C h_t, the selectivity innovati…
Read article →Math Capability Evals
How mathematical reasoning is measured in language models: the benchmark ladder from GSM8K to MATH to AIME and olympiad problems, answer-matching vers…
Read article →Matmul Arithmetic Intensity
The arithmetic intensity of matrix multiply from first principles: FLOPs/byte = 2MNK / bytes moved, how M, N, and K set data reuse, the roofline ridge…
Read article →Matryoshka Embeddings
How Matryoshka Representation Learning (MRL) trains embeddings so nested prefixes — the first 64, 128, 256, ... dimensions &…
Read article →Medusa Speculative Heads
A deep dive on Medusa speculative decoding: the extra heads as residual blocks, the conditional-independence gap that makes them decay with distance, …
Read article →Medusa
The math of Medusa decoding: bolting multiple lightweight residual heads onto a frozen model so each predicts a future token, expanding their top-k ou…
Read article →LLM Memory Math
How to compute the memory a language model needs to train: the four buckets (parameters, gradients, optimizer state, activations), why mixed-precision…
Read article →Mesa-Optimization
Mesa-optimization explained: the distinction between the base optimizer that trains a model and a learned optimizer that can emerge inside it, mesa-ob…
Read article →Micro-Batch Size Selection
Micro-batch size as the unit of work that must fit in memory: how it drives peak activation memory, how it trades against GEMM/kernel efficiency, and …
Read article →Milvus Math + Architecture
The math and architecture behind Milvus: the disaggregated, log-first design that separates compute from storage, collections and partitions and segme…
Read article →Min-P Sampling
Min-p sampling explained from first principles: the relative threshold p_base * max_prob, why scaling the cutoff with the model's con…
Read article →Mirostat Sampling
Mirostat as a feedback controller for text generation: how it targets a fixed surprise value tau, models the token distribution as Zipfian, estimates …
Read article →Mixed-precision training math
An overview of mixed-precision transformer training: fp16/bf16 compute copies over an fp32 master-weight spine, why gradient updates round to zero in …
Read article →MoE math architecture
The core math of Mixture-of-Experts layers: the router softmax, top-K selection, the sparse FFN forward pass, active vs total parameters, sparse-vs-de…
Read article →MQA Math
Multi-Query Attention (MQA): all query heads share one key/value head, cutting the KV cache by a factor of h. The shapes and math, the KV-cache reduct…
Read article →Multi-Agent Communication
The token-cost math of communication between LLM agents: message passing, the context-budget tradeoff of shared versus private state, star / chain / f…
Read article →Multi-Agent Coordination
How multiple LLM agents coordinate: decomposing a goal into a task graph, allocating tasks to agents, orchestrator/worker versus decentralized pattern…
Read article →Multi-Agent Debate
Multi-agent debate as a truthfulness and accuracy mechanism: several model instances argue, then a judge or a vote decides. Why debate can amplify cor…
Read article →Multi-Agent Emergence
Emergent collective behavior in multi-agent LLM systems: the analogy to swarms and complex adaptive systems, how division of labor and roles arise wit…
Read article →Multi-Agent Negotiation
Negotiation mechanics for LLM agents: utility functions and reservation values, the bargaining zone (ZOPA), offer and counter-offer protocols, Pareto-…
Read article →Multi-Agent Planning
How a team of LLM agents plans together: joint versus individual plans, the hierarchical planner-executor split, decomposing a goal into subgoals, a s…
Read article →Multi-Agent Reflection
Reflection in multi-agent systems: self-critique and cross-agent critique loops, the Reflexion pattern of verbal reinforcement, actor-evaluator-self-r…
Read article →Multi-Agent Shared Memory
Shared memory in multi-agent LLM systems: the blackboard / shared-state architecture versus message passing, consistency and staleness, memory as shar…
Read article →Multi-Agent System Theory
A unifying theory of multi-agent LLM systems: when several agents beat one, the decomposition-versus-overhead tradeoff, a taxonomy tying communication…
Read article →Multi-Hop RAG
Multi-hop retrieval-augmented generation: why one retrieval fails on compositional questions, query decomposition into sub-questions, the iterative re…
Read article →Multi-Head Attention
Multi-head attention as a structural choice: split d_model into h heads of d_model/h dimensions, run scaled dot-product attention in each independentl…
Read article →Multilingual Scaling
How languages scale together inside one model: the curse of multilinguality and capacity dilution, positive cross-lingual transfer versus interference…
Read article →Multilingual Tokenization
Why the same sentence costs 2-4x more tokens in some languages than in English: tokenizer fertility and tokens-per-word disparity across scripts, the …
Read article →Multimodal Scaling
Multimodal scaling laws: per-modality power laws, competition and synergy between modalities under a shared parameter and compute budget, the data-mix…
Read article →Muon optimizer architecture
Deep-dive on the Muon optimizer: momentum, orthogonalizing 2D weight updates toward U V^T, the quintic Newton-Schulz iteration, RMS-to-RMS shape scali…
Read article →muP
muP as a parametrization: the per-layer initialization-variance and learning-rate scaling rules that keep every activation, logit, and gradient update…
Read article →Neural Scaling Law Theory
Why neural scaling laws are power laws, not just that they are: the data-manifold-dimension theory (Sharma & Kaplan, alpha = 4/d where…
Read article →Neural Tangent Kernel
An intuitive overview of the Neural Tangent Kernel: why very wide neural networks behave almost linearly in their parameters, how training turns into …
Read article →Neural Tangent Kernel
A formal treatment of the neural tangent kernel: the definition Theta(x,x') = <grad_theta f(x), grad_theta f(x&amp…
Read article →NTK-Aware Interpolation
NTK-aware RoPE scaling explained from first principles: why rotary frequencies break outside the training window, the spectral-bias argument for leavi…
Read article →NVMe Offload Math
The arithmetic of NVMe/SSD offload for training giant transformers: how ZeRO-Infinity spills parameters and optimizer state to solid-state disk, the c…
Read article →Offload Math
The general framework for offloading model state off the GPU: the GPU-HBM to CPU-DRAM to NVMe memory-tier hierarchy, the overlap-vs-stall condition t_…
Read article →OpenAI Scaling Law Contributions
OpenAI's broader scaling program beyond the Kaplan power laws: GPT-2 zero-shot task transfer, the compute-efficient frontier, why lar…
Read article →OpenSearch Vector Search
The math behind OpenSearch vector search and the k-NN plugin: the knn_vector field, the three engines (Lucene, nmslib, Faiss) and when each wins, spac…
Read article →Optimized Product Quantization
Optimized Product Quantization (OPQ): why PQ's fixed subspace split is suboptimal when variance is unevenly spread across dimensions,…
Read article →Optimizer State Offload Math
The math of optimizer-state offload: why Adam's momentum and variance are the biggest, coldest bucket of training memory, how ZeRO-Of…
Read article →Optimizer State Memory Math
Optimizer-state memory in transformer training: the bytes-per-parameter for SGD, momentum, and Adam, why the fp32 master copy plus m and v add up to 1…
Read article →Optimizer Math for LLM Training
A unifying overview of the optimizers used to train transformers: the general update template, the SGD to momentum to RMSProp to Adam progression, ada…
Read article →PagedAttention architecture
Deep-dive on PagedAttention, the KV-cache management scheme behind vLLM that treats the cache like OS virtual memory. Covers why KV-cache fragmentatio…
Read article →Paged Attention Deep Dive
A deep dive on PagedAttention: how vLLM stores the KV cache in fixed-size blocks with a per-sequence block table, eliminates internal and external fra…
Read article →PagedAttention
Why naive contiguous KV-cache allocation wastes 60-80% of GPU memory to internal and external fragmentation and max-length over-reservation, and how P…
Read article →Paged KV Cache Deep Dive
A deep dive on the paged KV-cache block manager: choosing block size (tokens per block) and its fragmentation-vs-overhead trade-off, the block table a…
Read article →PEFT Theory Overview
The unifying theory behind parameter-efficient fine-tuning: why full fine-tuning is over-parameterized, the intrinsic-dimension hypothesis, the additi…
Read article →Persuasion Capability Evals
How to measure a language model's persuasion capability: the estimand of attitude and belief shift, randomized human-subject designs,…
Read article →pgvector Math + Architecture
The math and architecture of pgvector, the Postgres vector extension: the vector type, the L2 / cosine / inner-product operator classes, exact versus …
Read article →Pinecone Math + Architecture
A focused look at Pinecone, the managed vector database: serverless vs pod-based architecture, namespaces, the black-box managed ANN index, metadata f…
Read article →Pipeline Parallel Math Deep
Pipeline parallelism for transformers: splitting layers into stages across GPUs, the pipeline bubble of idle time under naive execution, microbatching…
Read article →Polysemanticity
Polysemanticity: why a single neuron in a transformer responds to many unrelated features, how it differs from its cause (superposition), how to measu…
Read article →Position Interpolation Math
Linear Position Interpolation (PI) for RoPE, derived from first principles: why direct extrapolation puts rotation angles out of distribution, the m -…
Read article →Positional Encoding
Why self-attention is blind to word order, and how positional encoding fixes it: permutation invariance, the absolute sinusoidal construction and its …
Read article →PP Math
The math of pipeline parallelism: splitting transformer layers into stages, the pipeline bubble fraction (p-1)/(m+p-1), GPipe versus 1F1B scheduling, …
Read article →PPO for LLMs Math
How PPO drives RLHF for LLMs: framing generation as a token-level RL problem, the reward-model-minus-KL-penalty reward, GAE advantage estimation, the …
Read article →PQ Math
The math of Product Quantization (PQ): splitting a vector into m sub-vectors, learning a k-means codebook per subspace, encoding a vector as m codewor…
Read article →Prefill Math
The math of the prefill phase: processing the whole prompt in one parallel forward pass to populate the KV cache and emit the first token, why prefill…
Read article →Prefix Tuning Theory
Prefix tuning explained from first principles: trainable prefix key and value vectors prepended to attention at every transformer layer, the reparamet…
Read article →Pre-norm vs post-norm transformers
Deep-dive on transformer normalization placement: post-norm on the residual path versus pre-norm inside the branch, gradient flow and attenuation, war…
Read article →Pretraining Scaling
How pretraining loss falls with compute, the C = 6ND accounting identity, planning a pretraining run from a fixed FLOP budget, token budgets and data …
Read article →Priority Scheduling
Priority scheduling for LLM serving: priority classes and strict priority queues, weighted fair queuing, preemption of low-priority requests and KV-ca…
Read article →Linear Probes
Probing classifiers for interpretability: linear and MLP probes trained on frozen activations to test what information is linearly decodable, why prob…
Read article →Prompt Prefill vs Decode
The decode phase in depth: why generating one token at a time is memory-bandwidth-bound with arithmetic intensity near 1, the per-token latency formul…
Read article →Prompt Lookup Decoding Math
The math of prompt lookup decoding (PLD): a zero-model speculative method that drafts from the prompt itself. How n-gram matching against the context …
Read article →Prompt Tuning Theory
Prompt tuning from first principles: relaxing discrete prompt search into a continuous soft-prompt matrix P: [p, d], where the gradient actually flows…
Read article →Qdrant Math + Architecture
The math behind Qdrant: collections, shards and segments; cosine as normalize-then-dot; the HNSW m / ef_construct / hnsw_ef budget and its memory bill…
Read article →QLoRA Theory
The theory behind QLoRA: why NF4 is a quantile grid rather than a uniform one, how double quantization recovers ~0.37 bits per parameter, the differen…
Read article →Quantization Quality Evaluation
How to evaluate a quantized LLM: perplexity delta and why it hides behavior changes, downstream task accuracy, per-layer error norms (MSE), signal-to-…
Read article →Advanced RAG Techniques Overview
The math of advanced RAG: query rewriting and multi-query expansion, why sparse and dense scores cannot be added directly, reciprocal rank fusion work…
Read article →Reasoning Capability Evals
The statistics of reasoning benchmarks: why GPQA Diamond, MMLU-Pro, ARC-AGI and BBH are structurally small, chance-corrected accuracy and the guessing…
Read article →Reasoning Model Scaling
How reasoning accuracy scales with chain-of-thought length and sampling: self-consistency (majority vote over K sampled chains) and its accuracy-vs-K …
Read article →Rectified Flow Math
Rectified flow from first principles: the linear interpolation between noise and data, the conditional-expectation velocity field and its L2 regressio…
Read article →Redis Vector Search Math
The math behind Redis vector search and RediSearch: inverted-index posting-list memory, TF-IDF and BM25 scoring with a worked example, why cosine, inn…
Read article →Representation Learning Theory
The math of learned representations: the InfoNCE objective and its log N mutual-information bound, temperature, the alignment-uniformity decomposition…
Read article →Reranking Math
The retrieve-then-rerank pipeline explained from the math up: dense bi-encoder first-stage retrieval, the cross-encoder that jointly encodes query and…
Read article →RetNet
Retentive Networks (RetNet) explained from first principles: retention as a fading-memory replacement for softmax attention, the recurrent form with a…
Read article →RetNet Math
The math of RetNet (Retentive Network): retention as a decay-weighted alternative to softmax attention, its three equivalent forms (parallel for train…
Read article →Retrieval Architectures Overview
Retrieval fundamentals from first principles: the bi-encoder factorization that turns ranking into vector search, embedding geometry, why dot product,…
Read article →Reward Hacking Math
The math of reward over-optimization in RLHF: Goodhart's law, the KL-regularized objective and its Gibbs-tilted optimum, the empirica…
Read article →Reward Model Math
How a reward model is trained for RLHF: bolting a scalar value head onto a pretrained language model, the Bradley-Terry preference model, the pairwise…
Read article →Reward Model Scaling
Reward normalization and scaling in RLHF: why raw reward-model scores need whitening before the PPO update, running mean/std normalization, per-batch …
Read article →Ring Attention
Ring Attention as a distributed systems problem: the activation and KV-cache memory bill at million-token context, where context parallelism sits amon…
Read article →Ring Attention Math Deep
Ring Attention explained from the math up: sharding a long sequence across P devices, keeping each query block fixed while K/V blocks rotate around a …
Read article →RLHF Math Deep Dive
A deep look at the RLHF objective itself: the KL-regularized reward-maximization objective max E[r(x,y)] - beta*KL(pi || pi_ref), why the KL leash is …
Read article →RLHF Scaling Laws
How RLHF results scale with policy-model size, reward-model size, and preference-data volume; the reward-model overoptimization scaling law from Gao, …
Read article →RLHF Scaling Math
The quantitative side of scaling RLHF: why sqrt(KL) is the natural coordinate for the reward-vs-drift frontier (a second-order Fisher argument), the b…
Read article →RMSNorm architecture - root-mean-square normalization in modern transformers
Deep-dive on RMSNorm: the root-mean-square formula, contrast with LayerNorm's mean-centering and bias, forward and backward passes, pre-norm resi…
Read article →RMSNorm Deep Dive
A deep dive on RMSNorm below the formula: the Jacobian as a rank d-1 projector, why the input gradient is exactly orthogonal to the input, a worked ba…
Read article →Roofline Model for LLM Ops
The roofline model applied to LLM inference: building the compute and bandwidth ceilings for a real machine, computing the ridge point, plotting a pre…
Read article →Rotary Position Embedding (RoPE)
The mechanism of Rotary Position Embedding, derived from scratch: the 2-D rotation-matrix and complex-exponential formulations, a proof that the inner…
Read article →Rotary Position Embedding Architecture in Depth
Positional encoding as a design problem: what a scheme must provide, how learned absolute, sinusoidal, T5 relative bias, ALiBi and rotary compare, whe…
Read article →RWKV Math
The math of RWKV: token shift as a two-tap causal convolution, the WKV operator as decay-weighted attention without QK^T, why it factorizes into a con…
Read article →Sparse Autoencoders (SAE)
Sparse autoencoders for transformer interpretability: the superposition hypothesis and why features outnumber dimensions, the encoder/decoder architec…
Read article →Sampling mathematics
The unified math of LLM decoding: softmax shift invariance, temperature as a rescaling of log-odds, the proof that entropy rises monotonically with te…
Read article →Scaled Dot Product Attention
Scaled dot-product attention worked out quantitatively: the variance derivation behind the 1/sqrt(d_k) factor, what softmax saturation does to gradien…
Read article →Scaling law architecture
How to use scaling laws as a design tool: the compute-optimal rule of thumb (C = 6ND, ~20 tokens per parameter), why compute-optimal is the wrong targ…
Read article →Scaling Laws
The mathematics of neural scaling laws: why loss follows a power law rather than an exponential, the joint form L(N, D) = E + A/N^alpha + B/D^beta and…
Read article →Scaling Laws
Where neural scaling laws break down: data-constrained scaling and the decaying value of repeated tokens, the effective-data model, mixture-of-experts…
Read article →Hyperparameter Scaling Transfer
Hyperparameter transfer across model scale from first principles: why the optimal learning rate drifts with width under standard parametrization, the …
Read article →ScaNN
How ScaNN accelerates maximum inner-product search with anisotropic, score-aware vector quantization: why the residual component parallel to a datapoi…
Read article →Score Matching
Score matching from first principles: why the normalizing constant makes maximum likelihood intractable, what the score function &nabl…
Read article →Search Scaling in Reasoning
Verifier-guided search as a test-time scaling axis: Best-of-N with a reward model and the coverage ceiling 1-(1-p)^N, why flat sampling wastes compute…
Read article →Selective Activation Recomputation
Selective activation recomputation from first principles: the ratio of FLOPs to rebuild a tensor over bytes freed by dropping it, why that ratio equal…
Read article →Self-Consistency Scaling
The math of self-consistency: majority voting over k independently sampled chains of thought. Derives the vote-accuracy curve, shows why the real bar …
Read article →Self-RAG
Self-RAG from first principles: reflection tokens as a learned vocabulary, the retrieve-or-not decision and when retrieval actively hurts, per-passage…
Read article →SentencePiece Deep Dive
SentencePiece as a complete tokenization system: raw text in and integer ids out with no pre-tokenization, the whitespace meta-symbol and the lossless…
Read article →Sequence Length Math
The cost math of sequence length in a transformer: the quadratic attention term versus the linear projection and FFN term, the crossover at s = 6d, th…
Read article →Sequence Parallel Math
Sequence parallelism explained with the math: partitioning activations along the sequence dimension to cut activation memory, the Megatron-style SP th…
Read article →Shampoo
How Shampoo works: the full-matrix AdaGrad ideal and why it is intractable, Kronecker factorization of the preconditioner into left and right factors,…
Read article →Shared KV Cache
Cross-layer KV sharing: cutting the KV cache by reusing keys and values across transformer LAYERS, not just across heads. How Cross-Layer Attention (C…
Read article →Signal Propagation in Deep Networks
Signal propagation theory for deep networks: the variance recursion through a layer and its fixed point, the mean-field order-chaos phase transition a…
Read article →Sliding Window Attention
The math of sliding-window attention (Mistral-style): each token attends only to the previous w tokens, cutting attention from O(N^2) to O(N*w) and ca…
Read article →SLO-Aware Scheduling Math
The queueing and scheduling math behind LLM latency SLOs: TTFT versus TPOT, prefill/decode interference, sizing a chunked prefill, utilization versus …
Read article →SmoothQuant Math
The math of SmoothQuant: enabling INT8 weight-and-activation (W8A8) quantization by migrating quantization difficulty from activations to weights with…
Read article →Softmax numerics architecture
How softmax is actually computed: the overflow range of fp16 and bf16, the max-subtraction identity and its proof, why the sum is accumulated in fp32,…
Read article →Sophia Optimizer
How Sophia builds a cheap diagonal curvature preconditioner for language-model pre-training: the clipped second-order update rule, the Hutchinson and …
Read article →SP Math
Sequence parallelism as a configuration decision: what the SP knob actually buys, why its degree is welded to the tensor-parallel degree, how it compa…
Read article →Sparse Retrieval Math
Sparse retrieval from first principles: the bag-of-words vector and the inverted index, TF-IDF and the full BM25 formula with term-frequency saturatio…
Read article →Speculative decoding architecture
Deep-dive on speculative decoding: exploiting memory-bound decode with a cheap draft model whose K proposals the target verifies in one parallel pass,…
Read article →Speculative Decoding in Practice
A systems deep dive on speculative decoding: choosing a draft model and the tokenizer-mismatch trap, self-speculation with Medusa heads and EAGLE feat…
Read article →Speculative Decoding
The probability theory of speculative decoding: the modified rejection sampling rule, a proof that the output distribution is exactly the target distr…
Read article →SPLADE Math
The math of SPLADE learned sparse retrieval: projecting hidden states through the masked-language-model head into vocabulary space, the log(1 + ReLU(w…
Read article →State Space Models Math
The math of linear time-invariant state space models (SSMs): the continuous system x' = Ax + Bu, y = Cx + Du, zero-order-hold and bil…
Read article →Selective SSM
How selective state space models work: making Delta, B and C functions of the input so the recurrence becomes time-varying, per-token ZOH discretizati…
Read article →Stable Diffusion Math
How Stable Diffusion works as a system: latent diffusion and the compute arithmetic that justifies it, the VAE encoder/decoder and the 0.18215 scale f…
Read article →State Space Models
The state space model (SSM) foundation behind S4, S4D and Mamba: the continuous linear system x'(t) = Ax(t) + Bu(t), ZOH and bilinear…
Read article →Stochastic Rounding
Stochastic rounding explained for low-precision training: round up with probability equal to frac(x/ulp) so the rounded value is unbiased, E[round(x)]…
Read article →Superposition Hypothesis
The superposition hypothesis from first principles: features as directions, the toy model of sparse features through a bottleneck, the Johnson-Lindens…
Read article →SwiGLU
How SwiGLU works: the gated feed-forward variant used in Llama and PaLM, FFN(x) = (Swish(xW) ⊙ xV)W2, why the gate needs three w…
Read article →Switch Transformer Math
Switch Transformer math: top-1 routing and why k=1 still gives the router a gradient, the expert capacity formula and token dropping, the product-of-f…
Read article →Synthetic Training Data
The mathematics of training on model-generated data: the recursive model-collapse recursion and a worked variance-shrinkage example, tail loss from fi…
Read article →Temperature Sampling Math
How sampling temperature acts on generation quality, measured rather than asserted: the asymmetric convergence rates at the cold and hot limits, the e…
Read article →Temperature scaling in LLM sampling
Deep-dive on temperature scaling: dividing logits by a scalar T before softmax to rescale logit differences, the cold/neutral/hot regimes and their ef…
Read article →Tensor Parallel Math Deep
A first-principles derivation of Megatron-style tensor parallelism: splitting a single matmul across GPUs with column-parallel then row-parallel linea…
Read article →Tensor Programs (Yang’s Framework)
Tensor Programs explained: what a tensor program is, what the master theorem says about infinite-width limits, the abc-parametrization of init scale, …
Read article →Test-Time Compute Math
Test-time (inference-time) compute scaling: spending more compute at inference to get better answers. The four ways to spend it -- longer chains of th…
Read article →tiktoken Deep Dive
How tiktoken actually works: byte-level BPE with 256 atoms and no unknown token, the pre-tokenization split regex used by the GPT-2, cl100k and o200k …
Read article →Top-K Sampling Math
The math of top-k sampling: the fixed-cardinality truncation rule and its renormalization, the k-th logit as an order statistic, why a fixed k imposes…
Read article →Top-P (Nucleus) Sampling Math
Nucleus sampling from first principles: the smallest-prefix cumulative-mass rule and its boundary token, why retained mass overshoots p and by how muc…
Read article →Transfer Learning Scaling
Transfer and downstream scaling laws: how pretraining scale turns into fine-tuning and downstream performance, the effective-data-transferred power la…
Read article →Typical Sampling
Locally typical sampling derived from first principles: the typical set and the asymptotic equipartition property, the uniform-information-density arg…
Read article →Unigram LM Tokenizer
The unigram language model tokenizer as a probabilistic model: the generative story of a sentence as independently drawn subword pieces, marginalizing…
Read article →Variance Maintenance in Deep Nets
The practical engineering of variance maintenance in deep transformers: what to instrument (per-layer activation RMS, the residual-stream growth curve…
Read article →Vector DB Math
How a vector database actually works: embedding storage and its memory cost, the ANN index (HNSW / IVF / PQ as pointers), distance metrics (cosine, do…
Read article →Verifier Scaling
Verifier-guided and best-of-N scaling: the accuracy-vs-N curve, why a good verifier beats majority vote, outcome vs process reward models, verifier-gu…
Read article →Vespa Math
The math that makes Vespa different from a vector store: first-class tensor fields and tensor ranking expressions, the multi-phase ranking cost equati…
Read article →LR Warmup Deep Dive
Why the first few hundred steps of transformer training need a smaller step size: Adam's thin second-moment estimate and what bias co…
Read article →Weaviate Math + Architecture
The arithmetic behind Weaviate: what a collection stores per object, named vectors, the dynamic-ef formula, thresholds measured in objects rather than…
Read article →Width Scaling
How transformer quality and training dynamics change as you widen d_model at fixed depth: the quadratic parameter law (about 12 * L * d^2 non-embeddin…
Read article →YaRN + NTK Scaling
YaRN explained from first principles: RoPE frequencies and wavelengths, why linear position interpolation crushes the high-frequency dimensions, the N…
Read article →ZeRO Stage 1 Math
How ZeRO Stage 1 partitions Adam optimizer states across data-parallel ranks: the 16-bytes-per-parameter memory model, why the fp32 master weights, mo…
Read article →ZeRO Stage 2 Math
ZeRO Stage 2 explained with the memory math: partition gradients as well as optimizer states so each rank keeps only its gradient shard, cutting gradi…
Read article →ZeRO Stage 3 Math
The memory and communication math of ZeRO Stage 3: full parameter partitioning that puts 1/N of every parameter, gradient, and optimizer state on each…
Read article →Zero-bubble pipeline parallelism
Deep-dive on zero-bubble pipeline scheduling: the (p-1)/m bubble, GPipe vs 1F1B vs interleaved 1F1B, splitting backward into input-gradient B and weig…
Read article →ZeRO-Infinity
ZeRO-Infinity as a synthesis: the arithmetic-intensity threshold that decides which memory tier can hold which state, bandwidth-centric partitioning a…
Read article →ZeRO Math
ZeRO (Zero Redundancy Optimizer) explained from the memory model up: the redundancy in vanilla data parallelism where every GPU holds full parameters,…
Read article →ZeRO-Offload
ZeRO-Offload explained with the math: why optimizer states dominate the 16-bytes-per-parameter Adam memory budget, how moving the fp32 states, the fp3…
Read article →Token Embedding Lookup
How the token embedding matrix works: the vocab x d_model lookup table, why a row lookup equals one-hot times E, the shapes from token ids to [N, d_mo…
Read article →Tokenizer Math
A first-principles deep-dive on tokenizer math: why subword tokenization sits between characters and words, the frequency-based BPE merge algorithm wi…
Read article →Training Data for SLMs
How to choose training data for a small language model: the quality-over-quantity thesis behind Phi's 'textbooks&…
Read article →Weight Initialization
Why weight initialization matters: variance preservation through layers to avoid vanishing and exploding activations and gradients, the forward and ba…
Read article →How Weights Are Stored on Disk
How model weights are stored and laid out in memory: row-major vs column-major order, strides and address arithmetic, why layout decides cache localit…
Read article →