All 44 articles, sorted alphabetically
Attention Is All You Need, in depth: the 2017 Transformer read closely, its exact training recipe, the scaling and complexity arguments, the ablations, and what modern models kept
A close reading of Vaswani et al. 2017: the post-norm encoder-decoder as specified, why attention is scaled by 1/sqrt(d_k) with numbers, a faithful Py…
Read article →Attention Is All You Need, in depth: a modern summary that follows one sentence through the encoder-decoder, the three masks, teacher forcing versus decoding, and a toy model you can train
A working engineer's summary of the 2017 Transformer paper: its five core claims, why recurrence was the bottleneck, one sentence traced through …
Read article →Attention Variants in 2026, in depth
A field guide to production attention in 2026: GQA, MLA, local-global interleaving, DeepSeek's indexer sparse attention, Gated DeltaNet hybrids a…
Read article →Causal vs Bidirectional Attention, in depth: the mask, the training objective it implies, KV caching, prefix LMs and the alignment bugs that leak the future
How causal and bidirectional self-attention differ: the mask on the score matrix, why next-token prediction needs a causal mask and masked language mo…
Read article →HuggingFace Datasets, in depth: publishing a training dataset on the Hub with explicit schemas, leak-free splits, Parquet shards, dataset cards and pinned versions
The producer side of Hugging Face Datasets: explicit Features, group-based splits that do not leak, exact deduplication, push_to_hub and Parquet shard…
Read article →Hugging Face Evaluate, in depth: metrics, comparisons, distributed accumulation and trustworthy model decisions
How the Hugging Face evaluate library works: metric, comparison and measurement modules loaded as code from the Hub, add_batch and the Arrow cache, di…
Read article →HuggingFace Hub, in depth: repositories and revisions, the local cache, pinning and offline loading, uploads and tags, Xet storage, tokens and gated models, and the trust model for weights and code
How the Hugging Face Hub works as infrastructure for models and datasets: git-backed repos with branches, tags and commit shas, the content-addressed …
Read article →HuggingFace Model Cards, in depth: the README.md metadata contract, what each key drives on the Hub, evaluation results, gating and generating cards in CI
How Hugging Face model cards work and how to write ones that are accurate: the YAML header and what license, library_name, pipeline_tag, base_model an…
Read article →Hugging Face PEFT, in depth: in-place injection, adapter files, loading, multi-adapter batches, hot-swapping and safe merging
How the Hugging Face PEFT library works at runtime: get_peft_model replacing layers in place with base_layer wrappers, LoraConfig fields that matter, …
Read article →HuggingFace Trainer, in depth: custom losses, num_items_in_batch and gradient accumulation, collators, column pruning and memory-safe evaluation
A practical guide to customizing the Hugging Face Trainer: what reaches the model after column pruning and collation, three ways to write a custom los…
Read article →Long-Context Evaluation, in depth: building a harness that measures effective context length
How to measure what a long-context model actually uses: retrieval, tracking, aggregation and multi-hop task families, needle, RULER and NoLiMa lessons…
Read article →Mixture of Depths, in depth: top-k token routing per block, the causality problem at sampling time, and what it costs to serve
How Mixture-of-Depths transformers spend compute unevenly across tokens: the router and top-k capacity per block, why multiplying by the router weight…
Read article →Rotary Position Embeddings (RoPE), in depth: the frequency ladder, what trained models do with it, testing it and choosing the base
RoPE beyond the derivation: the frequency ladder in numbers, how trained models split rotary channels between positional and semantic work, a property…
Read article →Relative Position Embeddings, in depth: Shaw, Transformer-XL, T5 buckets and DeBERTa, and what each costs in memory, kernels and the KV cache
How relative position embeddings work inside attention: Shaw et al.'s per-pair key and value vectors and the skewing trick that makes them afford…
Read article →The Transformer Breakdown: A Deep Dive into Self-Attention, Key-Value Pairs, and Positional Encoding
The Transformer architecture, introduced in the seminal 2017 paper "Attention Is All You Need," single-handedly revolutionized the field of Artificial Intelligence…
Read article →The Vanishing Gradient Problem: How Transformers Solved What Killed Earlier RNNs
In the early days of deep learning for sequence data—tasks like natural language processing or time series analysis—Recurrent Neural Networks (RNNs) were the…
Read article →Tokenization Choices Compared, in depth: base unit, algorithm, vocabulary size, pre-tokenization and how to measure them
A decision guide to tokenizer design for transformers: bytes versus characters, BPE versus WordPiece versus Unigram, the parameter and FLOP cost of vo…
Read article →Transformer Block Anatomy, in depth: the residual stream, attention and MLP sublayers, shapes, parameter and FLOP counts, and the bugs that hide in a block
A complete walk through one modern decoder-only transformer block: pre-norm residual stream, RMSNorm, grouped-query attention with RoPE, the SwiGLU ML…
Read article →Transformer Training Loss Curves, in depth: measuring loss correctly, comparing runs fairly, forecasting the end and alerting on trouble
The measurement side of transformer loss curves: token-weighted averaging across gradient accumulation and data-parallel ranks, label masking, tokens …
Read article →HuggingFace Accelerate Architecture, in depth: the Accelerator, prepare(), process state, data sharding, gradient accumulation, checkpoints and big-model loading
How Hugging Face Accelerate is built and how to use it well: the launcher and process state, what prepare() does to the model, optimizer, dataloader a…
Read article →HuggingFace Datasets Architecture, in depth: Arrow memory mapping, the fingerprint cache, streaming shards and distributed loading
How the Hugging Face datasets library actually works: Arrow memory mapping and why a 100 GB dataset opens instantly, load_dataset resolution, the map …
Read article →Hugging Face generate() architecture
Deep-dive on the Transformers generate() method: the GenerationConfig as single source of truth, the ordered logits-processor pipeline (temperature, t…
Read article →HuggingFace Transformers Library Architecture in Depth: loading, tokenizers, attention backends, generation and v5
How the Hugging Face Transformers library is built: Hub repos and Auto classes, PreTrainedModel and modular files, dtype and device_map memory math, t…
Read article →HuggingFace Transformers Pipelines Architecture in Depth
A 2500-word walkthrough of HuggingFace pipelines: task registry, model selection, preprocessor, forward, postprocessor, batching, device, streaming, O…
Read article →Tokenizers
Deep-dive on tokenizers: the normalization/pre-tokenization/subword pipeline, BPE/WordPiece/Unigram algorithms, vocabulary and token IDs, special toke…
Read article →Hugging Face Trainer architecture
Deep-dive on the HF Trainer: how TrainingArguments configures a run, the Trainer orchestrates the step over an Accelerate backend, gradient accumulati…
Read article →ALiBi attention architecture
Deep-dive on ALiBi (Attention with Linear Biases): replacing input position embeddings with a static per-head linear penalty on attention scores, the …
Read article →Attention sinks -- why the first tokens matter for streaming LLMs
Deep-dive on attention sinks: the infinite-generation KV-cache problem, sliding-window KV and its collapse, attention sinks (initial tokens absorbing …
Read article →Cross-attention architecture
Deep-dive on cross-attention, the attention variant where queries come from a target sequence and keys/values from a separate source, powering transla…
Read article →Dropout architecture, in depth: inverted dropout, where it sits in a transformer, variants, RNG under parallelism and when to turn it off
Dropout from first principles for transformer practitioners: inverted scaling and its variance cost, the embedding, attention and residual dropout sit…
Read article →FlashAttention -- IO-aware exact attention
Deep-dive on FlashAttention: the O(N^2) attention memory problem, the IO-awareness insight (attention is memory-IO-bound), tiling Q/K/V into SRAM bloc…
Read article →Transformer Gate Architecture, in depth: output gates on attention, gated cross-attention, residual scalars and decay gates in linear layers
Gating as a general mechanism in transformers: the sigmoid gate on the attention output and why it removes attention sinks, zero-initialised tanh gate…
Read article →KV cache architecture
Deep-dive on the transformer KV cache: prefill vs decode regimes, per-token memory math, paged allocation and block tables, prefix caching with copy-o…
Read article →MoE Transformer Architecture, in depth: where experts go, router design, shared experts, balancing, capacity and stability
The design space of a mixture-of-experts transformer block: which layers become MoE, coarse vs fine-grained and shared experts, token-choice vs expert…
Read article →Multi-head latent attention (MLA) architecture
Deep-dive on MLA: down/up-projecting keys and values into a cached low-rank latent, decoupled RoPE for position, weight absorption so inference attend…
Read article →Positional encoding architecture, in depth: absolute, relative, rotary, ALiBi and NoPE, and how to extend context
Why transformers need positional information and the main ways to provide it: sinusoidal and learned absolute embeddings, Shaw and T5 relative biases,…
Read article →Pre-norm vs post-norm architecture, in depth: where the norm sits, how it changes training, and how modern LLMs place it
An engineering guide to normalization placement in transformers: post-norm and pre-norm block code, the final norm, what each does to warmup and stabi…
Read article →QK-Norm architecture
Deep-dive on query-key normalization: why QK^T/sqrt(d) logits grow without bound, how softmax saturation kills gradients, per-head RMSNorm placement b…
Read article →Ring Attention: Architecture Deep-Dive
How ring attention shards sequences across devices and rotates KV blocks to compute exact attention beyond single-device memory limits — architecture,…
Read article →RMSNorm
Deep-dive on RMSNorm: normalizing activations by their root-mean-square with a learned gain and no bias, why dropping LayerNorm's mean-centering …
Read article →RoPE architecture, in depth: where rotary embeddings live in a transformer, layouts, caches and kernels
The engineering architecture of rotary position embeddings in a real transformer: where rotation is applied, the interleaved versus half-split layouts…
Read article →Sparse Attention Architecture, in depth: fixed patterns, block-sparse kernels, learned token selection and local/global layer mixes
How sparse attention works in practice: why dense attention becomes the bottleneck, fixed patterns (local, strided, global, random), block-sparse kern…
Read article →Speculative decoding architecture
Deep-dive on speculative decoding: why decoding is memory-bandwidth bound, a draft model proposing K tokens, single-pass parallel verification, the re…
Read article →SwiGLU architecture
Deep-dive on the SwiGLU FFN: the SiLU/Swish activation, the two up-projection gate multiplied elementwise and projected down, the two-thirds d_ff resc…
Read article →