EMBEDDINGS · ONE MATRIX, TWO JOBS
Tied embeddings: reuse the output weights
Look a token up in an 8-token embedding table, score every token with the same matrix, and size what tying saves in real models.
Your experiment
The tiny model has 8 tokens and 4 dimensions. Start with the input cat, 80% of its embedding left in the hidden state and a tied head, then lower that share, untie the head, change the token, and size the saving for each published model.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Lookup and scoring, step by step
Computed data
Metrics
The 8 x 4 embedding table, the separate output matrix and the context vector are small fixtures chosen to make the arithmetic readable; a real model transforms the hidden state through many layers before scoring. The savings table uses the published configs; Phi-3-mini-4k and Qwen2.5-7B ship untied, the others tied.
One matrix, two jobs
An embedding table E has one row per vocabulary token. At the input, looking up a token copies its row into the hidden state. At the output, the model scores every token by the dot product of the final hidden state with an output matrix. Tying uses E for both, so the model stores one V x d matrix instead of two. In the tiny model below, the input cat has the embedding 0.1, 0.9, 0.3, 0.
The input token wins
With the Share of the input embedding left in the hidden state at 80%, h is 0.11, 0.83, 0.2, 0.15, and the tied scores give cat 0.818, dog 0.761 and mat 0.535: cat scores itself highest because a vector has a large dot product with itself, REPEATS THE INPUT TOKEN. Lower the share to 0.3 and the context direction wins with mat at 0.6975. Clear Output head tied to the embedding and the separate output matrix scores on highest at 0.574. Real models need their layers to move the hidden state away from the input embedding before scoring.
What tying saves
With Size the saving for set to Gemma 2 2B, the shared table is 256,000 x 2,304 = 589,824,000 weights, 1.18 GB in bf16, which is 18.4% of what an untied model would hold (3,204,165,888). Qwen2.5-0.5B saves 136,134,656, or 21.6%; Llama 3.2 1B saves 17.5%. Phi-3-mini-4k ships untied and would save only 98,500,608, 2.6%, because its vocabulary is 32,064 tokens. Small models with large vocabularies gain most.
What tying gives up
A tied head cannot be stored at a different precision, pruned or resized separately from the input table, and both jobs pull on the same weights during training. Adding tokens to the vocabulary grows the shared matrix once instead of twice. Weight precision changes only the bytes: the Gemma 2 table is 2.36 GB in fp32 and 0.59 GB at 8 bits.
Reading the tool and its limits
The bars show the tiny model's logits, the text panel prints the lookup, the residual mix and every score, and the table sizes the shared matrix for each published configuration. The 8 x 4 matrices and the context vector are readable fixtures, not trained weights, and the hidden state is a simple mix rather than the output of transformer layers.