Encoder
1 · Input characters → token ids
2 · Embedding × √dmodel + positional encoding §3.4, §3.5
negative · 0 · positive. One row per character, one column per dimension.
3 · Self-attention softmax(QKᵀ/√dk)V
4 · Add & Norm → Feed-forward → Add & Norm max(0, xW₁+b₁)W₂+b₂
Decoder
5 · Outputs, shifted right
The decoder reads what it has written so far, starting from the start symbol.
6 · Masked self-attention future set to −∞
7 · Encoder–decoder attention Q from decoder, K and V from encoder
Output shared weights
8 · Linear + softmax §3.4
9 · Answer
This run vs the paper's base model
Why the paper dropped recurrence
An RNN must finish position 1 before it can start position 2, so its sequential work grows with the length of the input. Self-attention looks at every position in one matrix multiply. The paper's Table 1 makes this argument with big-O; here it is measured on the same laptop that trained the model.
Where the numbers come from
- The model is written by hand from the paper: scaled dot-product attention (eq. 1), multi-head attention, the position-wise feed-forward network (eq. 2), sinusoidal positions, post-norm Add & Norm, embeddings multiplied by √dmodel and shared with the output projection.
- It is trained with the paper's recipe: Adam with β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹, the warm-up learning-rate schedule (eq. 3) and label smoothing 0.1. Only the sizes are smaller, so it trains on a CPU.
- The page replays one forward pass in evaluation mode (dropout off). The answer is produced by greedy decoding, one character at a time; the attention shown for each step is the attention the model computed for that step.