All 18 articles, sorted alphabetically
The Transformer Breakdown: A Deep Dive into Self-Attention, Key-Value Pairs, and Positional Encoding
The Transformer architecture, introduced in the seminal 2017 paper "Attention Is All You Need," single-handedly revolutionized the field of Artificial Intelligence…
Read article →The Vanishing Gradient Problem: How Transformers Solved What Killed Earlier RNNs
In the early days of deep learning for sequence data—tasks like natural language processing or time series analysis—Recurrent Neural Networks (RNNs) were the…
Read article →Hugging Face generate() architecture
Deep-dive on the Transformers generate() method: the GenerationConfig as single source of truth, the ordered logits-processor pipeline (temperature, t…
Read article →HuggingFace Transformers Library Architecture in Depth
A 2500-word walkthrough of the HuggingFace Transformers library architecture: Hub, auto classes, tokenizer, model, Trainer, TRL/PEFT, Accelerate, and …
Read article →HuggingFace Transformers Pipelines Architecture in Depth
A 2500-word walkthrough of HuggingFace pipelines: task registry, model selection, preprocessor, forward, postprocessor, batching, device, streaming, O…
Read article →Tokenizers
Deep-dive on tokenizers: the normalization/pre-tokenization/subword pipeline, BPE/WordPiece/Unigram algorithms, vocabulary and token IDs, special toke…
Read article →Hugging Face Trainer architecture
Deep-dive on the HF Trainer: how TrainingArguments configures a run, the Trainer orchestrates the step over an Accelerate backend, gradient accumulati…
Read article →ALiBi attention architecture
Deep-dive on ALiBi (Attention with Linear Biases): replacing input position embeddings with a static per-head linear penalty on attention scores, the …
Read article →Attention sinks -- why the first tokens matter for streaming LLMs
Deep-dive on attention sinks: the infinite-generation KV-cache problem, sliding-window KV and its collapse, attention sinks (initial tokens absorbing …
Read article →Cross-attention architecture
Deep-dive on cross-attention, the attention variant where queries come from a target sequence and keys/values from a separate source, powering transla…
Read article →FlashAttention -- IO-aware exact attention
Deep-dive on FlashAttention: the O(N^2) attention memory problem, the IO-awareness insight (attention is memory-IO-bound), tiling Q/K/V into SRAM bloc…
Read article →KV cache architecture
Deep-dive on the transformer KV cache: prefill vs decode regimes, per-token memory math, paged allocation and block tables, prefix caching with copy-o…
Read article →Multi-head latent attention (MLA) architecture
Deep-dive on MLA: down/up-projecting keys and values into a cached low-rank latent, decoupled RoPE for position, weight absorption so inference attend…
Read article →QK-Norm architecture
Deep-dive on query-key normalization: why QK^T/sqrt(d) logits grow without bound, how softmax saturation kills gradients, per-head RMSNorm placement b…
Read article →Ring Attention: Architecture Deep-Dive
How ring attention shards sequences across devices and rotates KV blocks to compute exact attention beyond single-device memory limits — architecture,…
Read article →RMSNorm
Deep-dive on RMSNorm: normalizing activations by their root-mean-square with a learned gain and no bias, why dropping LayerNorm's mean-centering …
Read article →Speculative decoding architecture
Deep-dive on speculative decoding: why decoding is memory-bandwidth bound, a draft model proposing K tokens, single-pass parallel verification, the re…
Read article →SwiGLU architecture
Deep-dive on the SwiGLU FFN: the SiLU/Swish activation, the two up-projection gate multiplied elementwise and projected down, the two-thirds d_ff resc…
Read article →