Transformers

Attention Is All You Need

What aged and what didn't.

Attention Is All You Need

The 2017 paper that started everything, re-explained for 2026.

Attention Variants in 2026

MHA, MQA, GQA, MLA — what's deployed.

Causal vs Bidirectional Attention

Decoder encoder prefix-LM uses.

FlashAttention Explained

How the attention kernel was rewritten to be 5-10x faster.

HuggingFace Accelerate

How accelerate simplifies distributed training and inference across GPUs and nodes.

HuggingFace Datasets

How the datasets library provides memory-mapped, streamable access to huge datasets.

HuggingFace Evaluate

How the evaluate library provides standardized ML metrics (accuracy, BLEU, ROUGE, and hundreds more).

HuggingFace Hub

How the Hub distributes models, datasets, and Spaces, and how it centralized ML resource sharing.

HuggingFace Model Cards

What model cards are, why they matter for responsible ML, and how to write good ones.

HuggingFace PEFT

How the PEFT library implements LoRA, QLoRA, adapters, prefix tuning, and other efficient fine-tuning methods.

HuggingFace Pipelines

How pipelines provide task-focused inference APIs (classification, generation, etc.) with minimal setup.

HuggingFace Tokenizers Library

How the tokenizers library provides fast tokenization via Rust, and the algorithms it supports (BPE, WordPiece, Unigram).

HuggingFace Trainer

How the Trainer class abstracts training loops, handling optimization, evaluation, and checkpointing.

KV Cache Explained

Why generation is fast: the attention KV cache and its memory implications.

Long-Context Evaluation

Needle-in-haystack and what it misses.

Mixture of Depths

Routing tokens through different depths.

Multi-Token Prediction (MTP)

DeepSeek V3's MTP and the speed-quality trade.

Rotary Position Embeddings (RoPE)

The position encoding that replaced sinusoidal and enabled long-context extrapolation.

Relative position embeddings architecture

Deep-dive on relative position embeddings: distance, bias table, T5 buckets, Shaw edges, ALiBi, extrapolation.

The Transformer Breakdown: A Deep Dive into Self-Attention, Key-Value Pairs, and Positional Encoding

The Vanishing Gradient Problem: How Transformers Solved What Killed Earlier RNNs

Tokenization Choices Compared

BPE, SentencePiece, Tiktoken — and why it matters.

Transformer Block Anatomy

What's inside one of the N layers.

Transformer Training Loss Curves

Reading them and what each shape means.

HuggingFace Accelerate Architecture in Depth

A 2500-word walkthrough of HuggingFace Accelerate: Accelerator, config, DDP, FSDP, DeepSpeed, Megatron, TPU/MPS, mixed precision, save+resume.

HuggingFace Datasets Architecture in Depth

A 2500-word walkthrough of HuggingFace Datasets: Arrow storage, splits, cache, streaming, map/filter, Hub, data cards, interop.

Hugging Face generate() architecture

Deep-dive on the Transformers generate() method: the GenerationConfig as single source of truth, the ordered logits-processor pipeline (temperature, t…

HuggingFace Transformers Library Architecture in Depth

A 2500-word walkthrough of the HuggingFace Transformers library architecture: Hub, auto classes, tokenizer, model, Trainer, TRL/PEFT, Accelerate, and …

HuggingFace Transformers Library Overview

What the transformers library is, its role in the ML ecosystem, and the primitives it provides.

HuggingFace Transformers Pipelines Architecture in Depth

A 2500-word walkthrough of HuggingFace pipelines: task registry, model selection, preprocessor, forward, postprocessor, batching, device, streaming, O…

Tokenizers

Deep-dive on tokenizers: the normalization/pre-tokenization/subword pipeline, BPE/WordPiece/Unigram algorithms, vocabulary and token IDs, special toke…

Hugging Face Trainer architecture

Deep-dive on the HF Trainer: how TrainingArguments configures a run, the Trainer orchestrates the step over an Accelerate backend, gradient accumulati…

ALiBi attention architecture

Deep-dive on ALiBi (Attention with Linear Biases): replacing input position embeddings with a static per-head linear penalty on attention scores, the …

Attention sinks -- why the first tokens matter for streaming LLMs

Deep-dive on attention sinks: the infinite-generation KV-cache problem, sliding-window KV and its collapse, attention sinks (initial tokens absorbing …

Cross-attention architecture

Deep-dive on cross-attention, the attention variant where queries come from a target sequence and keys/values from a separate source, powering transla…

Dropout architecture

Deep-dive on dropout: standard, inverted, attention, residual, layer drop, DropPath, inference mode, modern LLM practice.

FlashAttention -- IO-aware exact attention

Deep-dive on FlashAttention: the O(N^2) attention memory problem, the IO-awareness insight (attention is memory-IO-bound), tiling Q/K/V into SRAM bloc…

Transformer gate architecture

Deep-dive on transformer gates: SwiGLU/GeGLU vs plain GeLU, MoE routing gate, attention output gating, numerics, kernel fusion.

KV cache architecture

Deep-dive on the transformer KV cache: prefill vs decode regimes, per-token memory math, paged allocation and block tables, prefix caching with copy-o…

MoE transformer architecture

Deep-dive on MoE transformers: gate MLP, top-K selection, expert MLPs, load balance loss, router z-loss, capacity factor, jitter.

Multi-head latent attention (MLA) architecture

Deep-dive on MLA: down/up-projecting keys and values into a cached low-rank latent, decoupled RoPE for position, weight absorption so inference attend…

Positional encoding architecture

Deep-dive on positional encoding: sinusoidal, learned, relative, RoPE, ALiBi, long-context extension (PI, NTK, YaRN), kernels.

Pre-norm vs post-norm architecture

Deep-dive on pre-norm vs post-norm placement: gradient flow, stability, LR warmup, DeepNet init, sandwich norm, final LN.

QK-Norm architecture

Deep-dive on query-key normalization: why QK^T/sqrt(d) logits grow without bound, how softmax saturation kills gradients, per-head RMSNorm placement b…

Ring Attention: Architecture Deep-Dive

How ring attention shards sequences across devices and rotates KV blocks to compute exact attention beyond single-device memory limits — architecture,…

RMSNorm

Deep-dive on RMSNorm: normalizing activations by their root-mean-square with a learned gain and no bias, why dropping LayerNorm&#x…

RoPE architecture

Deep-dive on RoPE and long-context extensions: rotation math, Position Interpolation, NTK-aware scaling, YaRN, kernel fusion, and eval.

Sparse attention architecture

Deep-dive on sparse attention patterns: local, strided, global, block, Longformer, BigBird, kernel support, training/serving.

Speculative decoding architecture

Deep-dive on speculative decoding: why decoding is memory-bandwidth bound, a draft model proposing K tokens, single-pass parallel verification, the re…

SwiGLU architecture

Deep-dive on the SwiGLU FFN: the SiLU/Swish activation, the two up-projection gate multiplied elementwise and projected down, the two-thirds d_ff resc…