MoE gate arithmetic and capacity accounting

Configure the router and experts

Inspect capacity and computation

Admission follows token-row order. Dropped routes are not rerouted or reweighted. Scroll each table horizontally for every column.

Compute sparse expert arithmetic

This experiment calculates actual gate probabilities, accepted expert evaluations and weighted sums for a small batch. Each input is one scalar token value. Each expert is a pair of slope and bias values and computes the maximum of zero and slope times input plus bias. The router logits are supplied independently of those scalars. They are not learned by this page. This allows you to isolate gate selection from expert arithmetic and capacity, without implying that a scalar ReLU replaces a trained Transformer feed-forward block.

Set every input deliberately

Token inputs accepts a JSON array of one through sixteen numbers. Router logits is a rectangular matrix with one row per token and one column per expert. Expert functions accepts one through eight rows, each exactly [slope, bias]. Every numeric entry must be finite and between minus one hundred and one hundred. Top-k chooses one through eight routes per token and must not exceed the actual expert count. Capacity slots per expert is an integer from zero through sixteen. Gate normalization chooses selected top-k or full softmax. Add input residual adds each original scalar to its accepted branch output. Example chooses Balanced requests, Crowded expert or Equal-logit ties; Load example replaces the three JSON editors but preserves capacity, top-k, normalization and residual. Reset defaults restores the balanced batch, top-one, two slots, selected normalization and no residual.

Follow the probability and weighting equations

The router subtracts the largest logit in each row, exponentiates the shifted values, and divides by their sum. This is numerically stable softmax. Top-k ranks the original logits, with smaller expert number breaking equal-score ties. Full softmax keeps the selected experts original probabilities. Selected normalization divides each selected probability by the sum over selected experts, making requested weights sum to one. Admission happens afterward. Each accepted contribution is its weight times its expert value; the branch output sums accepted contributions. Dropped routes do not execute an expert, and surviving weights are not normalized again. Residual addition is separate from both selection and admission.

Reproduce balanced outputs and weighted sums

Reset defaults. Inputs [1,2,3,4] and alternating router rows [2,0] and [0,2] choose alternating experts. Expert one is ReLU(x); expert two is ReLU(5 minus x). With top-one and selected normalization, outputs are exactly [1,3,3,1]. Full softmax multiplies each branch by exp(2) divided by exp(2) plus one, approximately 0.880797. Choose top-two and capacity four: both experts execute for all four tokens. The first output becomes 0.880797 times one plus 0.119203 times four, approximately 1.357609. For two experts, top-two includes the whole distribution, so selected and full normalization agree.

Separate imbalance from insufficient capacity

Load Crowded expert with top-one, capacity two, selected normalization and no residual. All four requests choose expert one. Only the first two tokens execute; the last two have branch output zero. Expert two still has two unused slots: spare capacity elsewhere is not an automatic fallback. Raising capacity to four removes dropped routes but leaves requested-load imbalance unchanged. With capacity zero no experts execute. Enable the residual and outputs equal the inputs, even though every route was dropped. Under top-two and capacity two, the first two tokens fill both experts and the last two are entirely unserved. Admission uses token array order, so changing that order can change which values are processed.

Inspect ties and conservation identities

Load Equal-logit ties, select top-one and capacity four. Both probabilities are one half, yet deterministic ties send all requests to expert one. Top-two requests both experts equally. For every setting, requested routes equal token count times top-k; accepted plus dropped equals requested. Accepted plus unused slots equals expert count times capacity. A token with zero admitted routes has zero branch output, even if its residual output is nonzero. Requested-load CV is population standard deviation divided by mean requests per expert; maximum / mean is a second imbalance ratio. These metrics use requests before capacity, so dropped routes cannot make a crowded router appear balanced.

Read exact evidence and respect the limits

The expert table reports requested, accepted, dropped and unused counts. The route table exposes full probability, selected weight, admission, expert output and contribution; a rejected expert value says Not executed. The token table reports probabilities, accepted route count, retained weight, branch sum and final output. Complete state evidence preserves unrounded JavaScript numbers. Tables are keyboard-focusable and scroll horizontally on phones. Invalid inputs clear all tables instead of leaving stale results. This direct arithmetic instrument has no animation clock or measured latency. It omits training, auxiliary balance loss, vector-valued experts, learned routers, distributed dispatch costs and production fallback policies. Neither balanced load nor a larger top-k establishes better model quality.