Why architecture matters here
Embeddings matter because they're the fundamental interface between discrete tokens and the continuous vectors the transformer reasons over -- at both the input and output -- and they encode the model's learned representation of tokens. A transformer operates on continuous vectors (attention, feed-forward -- all vector operations), but language is discrete tokens -- so embeddings are the essential interface: at the input, converting tokens to vectors (so the model can process them); at the output, converting the model's vectors back to token predictions (logits over the vocabulary). Without embeddings, there's no way to feed tokens into the vector-based model or get token predictions out. And the embeddings are learned representations (encoding the model's understanding of the tokens -- the embedding space's geometry capturing semantic relationships) -- so they're not just a mechanical conversion but a learned, meaningful representation. For understanding transformers (how tokens enter and predictions exit, and how tokens are represented), embeddings are fundamental, and understanding them (the lookup, the matrix, the learned geometry, weight tying, the output projection) is understanding the transformer's input/output interface.
The lookup-into-a-learned-matrix insight is the mechanical core, and it clarifies what an embedding is. A token embedding is conceptually simple: the embedding matrix is a learned matrix of shape vocabulary size × d_model (one row per vocabulary token -- each row a d_model-dimensional vector). To embed a token, you look up its row (the token's integer id indexes the matrix -- row id is the token's embedding vector) -- a simple lookup (no computation -- just indexing the matrix). So a token embedding is just the token's row in the embedding matrix (a learned vector per token). What makes it powerful is that the matrix is learned: through training, the model adjusts the embedding vectors (the matrix's rows) so that they're useful representations (the geometry of the space encoding meaning -- similar tokens' vectors near each other, meaningful directions). So the embedding matrix is a learned lookup table (mapping each token to its learned vector) -- the lookup is mechanical (indexing), but the vectors are learned (meaningful). This clarifies what an embedding is: a learned vector per token, retrieved by a simple lookup into the learned embedding matrix. Understanding the lookup-into-a-learned-matrix core (a token's embedding is its learned row in the embedding matrix) is understanding the mechanical and conceptual core of embeddings.
And the weight-tying-plus-output-projection insight connects the input and output, an elegant and important detail. The input embedding (tokens → vectors) and the output projection (the final hidden vector → vocabulary logits) are conceptually inverse operations (one maps tokens to vectors, the other maps vectors to token scores). The output projection is a matrix of shape d_model × vocabulary size (projecting the d_model hidden vector to a vocabulary-size vector of logits -- a score per token, for predicting the next token). Weight tying is the elegant technique of sharing the same matrix for both: the input embedding matrix (vocab × d_model) and the output projection matrix (d_model × vocab) are the same matrix (one is the transpose of the other) -- so the model has one shared matrix for embedding tokens in and projecting predictions out. This has benefits: it saves parameters (one matrix instead of two -- significant, since the embedding matrix is large -- vocab × d_model), and it often improves quality (the shared representation -- the input embedding and output projection using the same learned token representations -- a useful inductive bias). Weight tying is common in language models (a standard technique). So the output projection (vectors → logits) and weight tying (sharing the matrix with the input embedding -- saving parameters, improving quality) connect the input and output embeddings elegantly. Understanding weight tying and the output projection (the input and output embeddings, often sharing a matrix) is understanding an elegant and important detail of embeddings.
The architecture: every piece explained
Top row: the input embedding. Token embeddings: converting each token (an id) to a vector -- a lookup into the embedding matrix (the token's id indexes its row/vector). Embedding matrix: the learned matrix of shape vocabulary size × d_model (one d_model-dimensional vector per vocabulary token) -- the lookup table. Learned representations: the embedding vectors are learned (through training -- the geometry encoding meaning: similar tokens near each other, meaningful directions) -- the model's learned token representations. Positional info: combined with the token embeddings (so the model knows token order) -- added as positional embeddings, or applied via mechanisms like RoPE (rotary positional embeddings) -- adding position.
Middle row: output and details. Weight tying: sharing the same matrix for the input embedding and the output projection (the embedding matrix and its transpose) -- saving parameters (one large matrix instead of two) and often improving quality (shared representation). Output projection: projecting the final hidden state (d_model vector) to logits over the vocabulary (a score per token -- for next-token prediction) -- a matrix of shape d_model × vocabulary size. Scaling: a scaling factor (often √d_model) applied to the embeddings (a detail affecting the magnitude/training) -- the embedding scaling. Vocab size effects: the embedding/output matrices are vocab × d_model -- so a large vocabulary means many parameters (the embedding matrix large) and output compute (the logits over a large vocabulary) -- the vocabulary's cost.
Bottom rows: contrast and geometry. vs sentence embeddings: token embeddings (a vector per token -- the transformer's input interface) vs sentence embeddings (a vector per whole sequence -- a different use, e.g., for semantic search -- produced by pooling/encoding a sequence to one vector) -- token vs sequence embeddings (different granularity and use). Embedding space: the geometry of the learned embedding space -- similarity (similar tokens' vectors near each other -- e.g., by cosine similarity) and arithmetic (meaningful directions -- the classic word-vector arithmetic) -- the semantic structure. The ops strip: vocab design (designing the vocabulary -- the tokenizer -- affecting the vocabulary size, which affects the embedding parameters and output compute -- and the tokenization quality), tying (using weight tying -- sharing the input/output matrix -- saving parameters, improving quality -- a standard choice), and init (initializing the embeddings -- the initialization scheme affecting training -- and the scaling factor).
End-to-end flow
Trace tokens through the embedding interface. A sequence of tokens (integer ids) enters the model. Each token's id indexes the embedding matrix (the id-th row) -- retrieving its embedding vector (a d_model-dimensional learned vector) -- so the sequence of token ids becomes a sequence of embedding vectors (the input to the transformer layers). Positional information is combined (added positional embeddings, or RoPE applied in attention) -- so the model knows the token order. The transformer layers process these vectors (attention, feed-forward -- reasoning over the vectors). At the output, the final hidden state (a d_model vector per position) is projected to logits over the vocabulary (via the output projection matrix -- d_model × vocab -- producing a score per vocabulary token) -- and (for the next-token prediction) the logits are used (e.g., softmax to probabilities, then sampling) to predict the next token. So the embeddings converted the tokens to vectors (input -- the lookup), and the output projection converted the model's vectors back to token predictions (output -- the projection to logits) -- the input/output interface. With weight tying, the input embedding matrix and the output projection matrix are the same (shared) -- one matrix for both conversions.
The learned-representation and weight-tying vignettes show the depth. A learned-representation case: the team examines the embedding space -- finding that semantically similar tokens have nearby vectors (e.g., synonyms, related words -- close by cosine similarity) and that there are meaningful directions (the geometry encoding relationships) -- the learned representations capturing semantic structure (the model having learned, through training, to arrange the tokens meaningfully in the embedding space). The learned embeddings encoded meaning. A weight-tying case: the model uses weight tying (the input embedding and output projection sharing the matrix) -- saving parameters (one vocab × d_model matrix instead of two -- significant for a large vocabulary) and improving quality (the shared representation) -- the weight tying being a standard, beneficial choice. The weight tying saved parameters and helped quality.
The vocab-size and geometry vignettes complete it. A vocab-size case: the team considers the vocabulary size -- a larger vocabulary (more tokens) means a larger embedding matrix (vocab × d_model -- more parameters) and more output compute (logits over more tokens) -- but potentially better tokenization (fewer tokens per text -- shorter sequences). They balance the vocabulary size (the tokenizer design -- balancing the embedding/output cost against the tokenization quality) -- the vocabulary size affecting the model's parameters and compute. A geometry case: the team leverages the embedding space geometry (similarity -- for tasks using token similarity; the meaningful structure) -- the learned embedding space being useful beyond just the model's internal processing (e.g., the representations reflecting semantic relationships). The consolidated discipline the team documents: understand embeddings as the transformer's input/output interface (token embeddings -- a lookup into the learned embedding matrix, converting tokens to vectors; the output projection -- converting the final hidden state to vocabulary logits), recognize the learned representations (the embedding space's geometry encoding meaning -- learned through training), use weight tying (sharing the input embedding and output projection matrix -- saving parameters, improving quality -- a standard choice), account for the scaling factor and the vocabulary-size effects (a large vocabulary meaning more parameters and output compute -- balanced against tokenization quality), distinguish token embeddings from sentence embeddings (token vs sequence granularity), and design the vocabulary and initialization appropriately -- because embeddings are the fundamental interface between discrete tokens and the continuous vectors the transformer reasons over (at both input and output), encoding the model's learned representation of tokens, with weight tying elegantly connecting the input and output.