Why architecture matters here

Small MoE fails on expert collapse, routing overhead, and unsuitable serving. Architecture matters because SLM economics require careful expert count + routing.

Advertisement

The architecture: every piece explained

The top strip is design. Small dense base. Convert to MoE expert count + top-K. Router gate. Sparse activation.

The middle row is training. Load balance. Distillation from dense teacher. Serving fit small-cluster feasible. Metrics quality per FLOP.

The lower rows are ops. Eval. Failure modes. Ops — sizing + deploy + cost.

Small MoE — dense-to-sparse + expert count + routing + activation sparsitySLM capacity via sparse expertsSmall dense base1B-7BConvert to MoEexpert count + top-KRouterlearned gateSparse activationfew experts per tokenLoad balanceaux lossDistillationfrom dense teacherServing fitsmall-cluster feasibleMetricsquality per FLOPEvalper-task liftFailure modescollapse + overheadOps — sizing + deploy + costregularizedistillfitmeasureevaldetectdetectoperateoperate
Small MoE architecture: base + experts + routing.
Advertisement

End-to-end flow

End-to-end: 1B base + 8 experts + top-2 routing. Distilled from a 7B teacher. Load balance auxiliary loss keeps utilization even. Active params 2B; total 8B. Deployed on modest GPU; quality between 3B and 7B dense.