MoE: route, admit and combine

Configure the router and experts

Inspect capacity and computation

Admission follows token-row order. Dropped routes are not rerouted or reweighted. Scroll each table horizontally for every column.

Compute routing, then compute outputs

This experiment separates three decisions: which experts a token requests, which routes fit the available slots, and how accepted expert outputs combine. Each input token is one scalar. Each expert is a small actual function, ReLU(slope times input plus bias), using its editable pair of weights. Router logits are entered directly, one row per token and one column per expert. A stable softmax converts these numbers into probabilities. Top-k ranks the logits in descending order; exact ties choose the lower expert number first. There is no random routing, inferred linguistic specialization or learned router on this page.

Set the inputs deliberately

Example chooses Balanced requests, Crowded expert or Equal-logit ties. Selection alone preserves your edits; Load example copies the input, logit and expert arrays while preserving the other controls. Input accepts one through sixteen numbers. Experts accepts one through eight rows, each exactly [slope, bias]. Logits must match the token count by expert count. All entries must be finite JSON numbers between minus one hundred and one hundred. Top-k chooses one through eight requested experts per token and cannot exceed the entered expert count. Invalid dimensions, nonnumeric values or impossible k clear all old results. Reset defaults restores four tokens, two experts, top-one, capacity two, selected normalization and no residual.

Choose weights and admission rules

Gate normalization offers two explicitly different policies. Selected experts renormalizes the chosen probabilities to sum to one before admission. Full softmax keeps the original probabilities, so a top-one weight can be less than one. Capacity slots per expert is an integer from zero through sixteen. Tokens are admitted in input-row order; each expert accepts at most this many routes. Excess routes are dropped without rerouting, and remaining weights are never renormalized after a drop. Reordering tokens can therefore change who gets served. These are teaching choices, not a claim that every MoE architecture uses the same dispatcher. Add input residual adds the original scalar after the expert sum; it does not restore a rejected expert computation.

Reproduce a balanced computation

Reset defaults. Tokens are [1, 2, 3, 4], with alternating logits [2, 0] and [0, 2]. Expert one computes ReLU(x), and expert two computes ReLU(5 minus x). Top-one requests alternate between experts. Both receive two requests, and capacity two accepts all four routes. Selected normalization gives weight one, producing outputs [1, 3, 3, 1]. Requested-load coefficient of variation is zero and maximum divided by mean is one. Switch to Full softmax: every accepted weight is about 0.880797, reducing each branch output by that factor. This changes output weighting without changing the chosen experts or capacity decisions.

Expose overflow and starvation

Load Crowded expert with top-one, selected normalization and capacity two. Every token requests expert one. The first two execute, while tokens three and four have zero branch output. There are two dropped routes and two entirely unserved tokens, even though expert two has two unused slots. Requested-load CV is one and maximum divided by mean is two. Increase capacity to four: the outputs become [1, 2, 3, 4], but the requested-load imbalance remains. Set capacity zero and enable the residual: all four expert computations are skipped, yet final outputs equal the original inputs. A residual path preserves values; it does not prove the sparse branch processed them.

Compare top-two and ties

Reset, then select top-two and capacity four. Both experts execute for every token, so the eight requests all fit. Under selected normalization the first token combines approximately 0.880797 times one with 0.119203 times four, yielding approximately 1.357609. With capacity two, the first two tokens fill both experts and the last two lose both routes. Load Equal-logit ties and choose top-one: lower-numbered expert one receives every request despite uniform softmax probabilities. Switch to top-two to request both experts equally. Probability symmetry alone is not load balance when deterministic tie-breaking and sparse selection intervene.

Read the tables and limits

Per-expert rows separate requested, accepted, dropped and unused slots. Route rows show ranking, full probability, chosen weight, admission, actual expert value and weighted contribution; rejected routes say Not executed. Token rows expose all probabilities, retained gate mass, branch and residual output. Requested-load CV is the population standard deviation divided by mean request count; maximum divided by mean is another imbalance measure. Neither metric is a quality label. Complete state evidence retains unrounded numerical values, and tables scroll horizontally with keyboard focus on phones. This direct calculator needs no fabricated playback clock. It uses JavaScript double precision, tiny scalar experts and explicit finite bounds. It excludes training, auxiliary balancing loss, full FFNs, communication, batching across devices and hardware timings. Parameter and evaluation counts describe only these scalar functions, not model memory or inference speed.

Primary reading: Switch Transformers discusses sparse routing and finite expert capacity. This calculator's selectable top-k, fixed slot count, scalar functions and normalization policies are explicitly simplified choices.