Artificial Intelligence · Machine Learning Architecture

Mixture of Experts (Machine Learning)

Reference entry · last updated September 11, 2026

Mixture of Experts (MoE) is a machine learning architecture that partitions network computation across multiple sub-networks, designated as experts, with a gating mechanism directing inputs to specific subsets of those experts [1]. By executing conditional computation, MoE decouples total model parameter count from per-sample floating-point operations (FLOPs). In large-scale language models, replacing standard dense feed-forward network (FFN) layers with sparse MoE layers allows expanding model parameter capacity by orders of magnitude while preserving constant per-token inference FLOP budgets [3, 4].

1. First Principles and Mathematical Formulation

Standard dense neural networks process all inputs through every parameter in every layer. For an input representation \(x \in \mathbb{R}^d\), a dense feed-forward layer with intermediate dimension \(d_{ff}\) executes \(2 \cdot d \cdot d_{ff}\) multiply-accumulate operations per token. Under dense scaling laws, increasing parameter capacity requires increasing matrix dimensions, expanding FLOPs quadratically or linearly with width and depth.

A Mixture of Experts architecture relaxes this coupling by introducing conditional computation. Let \(\{E_i\}_{i=1}^N\) denote a collection of \(N\) distinct expert networks, each mapping \(\mathbb{R}^d \to \mathbb{R}^d\). A parameterized gating network \(G\) computes a routing distribution over the experts from input \(x\):

Here \(G(x)_i\) represents the routing weight assigned to expert \(i\), satisfying \(G(x)_i \ge 0\) and \(\sum_{i=1}^N G(x)_i = 1\). In a sparse MoE, \(G(x)\) is constrained such that only a small number of coordinates \(k \ll N\) are non-zero. The active computation evaluates only the \(k\) selected experts, preserving computational cost proportional to \(k\) rather than total capacity \(N\) [3].

Early formulations by Jacobs et al. (1991) and Jordan & Jacobs (1994) applied modular gating networks to supervised classification and regression, training local linear models via maximum likelihood and Expectation-Maximization [1, 2]. Modern deep architectures place sparse MoE modules within Transformer layers, substituting the point-wise feed-forward block while retaining dense multi-head self-attention.

2. Routing Mechanisms and Load Balancing

Noisy Top-k Gating

To enable end-to-end backpropagation through discrete routing choices, Shazeer et al. (2017) introduced Noisy Top-\(k\) Gating [3]. For an input \(x\), a linear transformation with weight matrix \(W_g \in \mathbb{R}^{d \times N}\) produces unnormalized routing logits. Gaussian noise is added during training to promote exploratory routing across experts:

The top \(k\) values are retained, with all remaining values set to \(-\infty\):

The routing gate coefficients are obtained by evaluating the Softmax function over the thresholded vector:

Expert Capacity and Token Dropping

Hardware accelerators (GPUs and TPUs) require fixed tensor shapes for static execution graphs and balanced communication. If tokens route unevenly, an over-subscribed expert creates a bottleneck, delaying synchronization across parallel devices.

To bound computational load per accelerator, implementations define an Expert Capacity limit [4, 5]. For a batch containing \(T\) tokens distributed across \(N\) experts, the maximum number of tokens processed by any single expert is governed by a capacity factor \(C\):

When \(C = 1.0\), an expert receives strictly its fair share of tokens under uniform distribution. If more than \(\text{Capacity}\) tokens route to a specific expert, the excess tokens are dropped: their representations bypass the expert layer via residual connections without modification, or route to alternate experts [4]. Setting \(C > 1.0\) provides a buffer against routing imbalance at the cost of zero-padded computation.

Auxiliary Load-Balancing Loss

Without explicit incentives, gating networks suffer from routing collapse, where a small fraction of experts receives all tokens while the rest remain unutilized. To encourage uniform distribution across all \(N\) experts, training objectives incorporate an auxiliary load-balancing loss [3, 4, 5].

For a batch of tokens \(\mathcal{B}\), define \(f_i\) as the fraction of tokens routed to expert \(i\), and \(P_i\) as the average routing probability assigned to expert \(i\) prior to top-\(k\) masking:

The auxiliary loss is proportional to the dot product of vectors \(f\) and \(P\):

The scalar hyperparameter \(\alpha\) weights the loss relative to the primary language modeling objective. Because \(P_i\) is continuous and differentiable with respect to gate parameters \(W_g\), gradients from \(\mathcal{L}_{\text{balance}}\) penalize gates that concentrate probability mass onto already crowded experts.

3. Architectural Variations in Large Language Models

Switch Routing (Top-1)

Fedus et al. (2022) proposed the Switch Transformer, which simplifies sparse routing by selecting exactly one expert per token (\(k = 1\)) [4]. When top-\(k\) routing applies a post-selection Softmax over the chosen subset, setting \(k = 1\) would trivially normalize the surviving weight to \(G(x)_i = 1.0\), eliminating gradient signals from the task loss through the router. Instead, the Switch Transformer computes the Softmax distribution over all \(N\) candidate experts first, selects the top-1 index \(i = \operatorname{argmax}_j (x \cdot W_g)_j\), and gates the selected expert output with its pre-selection softmax probability \(p_i(x)\):

Retaining the pre-selection probability preserves continuous gradients from task loss back into the router weights \(W_g\). Switch routing halves communication volume relative to top-2 architectures, eliminates intra-token expert blending, and scales model size to over one trillion parameters while maintaining training throughput.

Fine-Grained Segmentation and Shared Experts

Standard MoE implementations deploy moderate numbers of large experts (such as 8 or 16 experts with top-2 routing). Dai et al. (2024) introduced DeepSeekMoE, establishing that finer granularity improves knowledge specialization and parameter efficiency [6].

DeepSeekMoE decomposes each expert into \(m\) smaller experts, scaling expert count to \(m \cdot N\) while activating \(m \cdot k\) units per token. This configuration keeps parameter activation budgets identical while creating exponentially more combinatorial routing paths. In addition, the architecture isolates a subset of experts as dedicated shared experts that process every token unconditionally:

Here \(\mathcal{S}\) denotes the set of shared experts and \(\mathcal{R}\) denotes the routed subset. Isolating shared representations into non-routed parameters prevents redundant common-knowledge duplication across individual routed experts [6].

4. Systems, Hardware, and Serving Trade-Offs

Expert Parallelism

Because total parameter counts in MoE models exceed the memory capacity of single accelerators, models use Expert Parallelism (EP) [5]. Different experts reside on distinct physical accelerators across an interconnect fabric. Under EP, tokens undergo an All-to-All collective communication operation: each accelerator computes routing assignments for its local tokens, dispatches tokens to the designated remote expert accelerators, executes expert feed-forward computations locally, and runs a reverse All-to-All to return output tokens to their origin devices.

The communication latency of All-to-All transfers is bounded by cross-node network bandwidth (such as InfiniBand or NVLink). When communication overhead exceeds local computation time, training and inference efficiency degrades.

Serving Dynamics and Memory Bandwidth

Inference throughput in autoregressive generation is dominated by memory bandwidth during the single-token decoding phase. In dense models, every parameter is loaded from High Bandwidth Memory (HBM) into accelerator compute cores for every generated token. In sparse MoE models, only the weights of active experts are fetched. This yields substantial latency speedups when batch sizes are small.

However, total memory capacity requirements depend on total loaded parameters rather than active parameters. Serving an MoE model requires sufficient accelerator memory or Host RAM offload to store all \(N\) experts simultaneously, establishing distinct regimes for operational cost:

5. Empirical Properties and Failure Modes

Empirical analyses of MoE language models demonstrate systematic patterns and operational failure modes:

• Syntactic vs. Semantic Specialization: Gating decisions often correlate with lower-level syntactic classes, punctuation, and token domains rather than high-level conceptual disciplines.
• Transfer Learning Instability: MoE models can experience fine-tuning instabilities or router overfitting when downstream task datasets are small or domain-skewed, as gating distributions diverge from pre-training equilibrium.