Mixture of experts (MoE) is a machine learning technique in which multiple expert networks, or learners, divide a problem space into homogeneous regions, representing a form of ensemble learning. In such systems, a routing mechanism - often called a gating function or weighting function - determines how inputs are assigned to experts and how their outputs are combined. Mixture of experts routing thus refers to the specific algorithms and design choices that govern this selection and weighting process, which are critical for balancing model capacity, computational cost, and output quality. The concept has evolved from early statistical models in the 1990s to the sparse routing schemes used in contemporary large language models, where it enables massive parameter counts with relatively low per-token computation.
The core architecture of any mixture of experts system includes a set of expert functions \(f_1, ..., f_n\), each taking the same input \(x\) and producing an output, and a weighting function \(w\) that maps \(x\) to a vector of non-negative weights \((w(x)_1, ..., w(x)_n)\). The final output is typically computed as a weighted sum: \(f(x) = \sum_i w(x)_i f_i(x)\). Both the experts and the weighting function are trained jointly by minimizing a loss function, usually via gradient descent. The routing mechanism determines how these weights are computed, whether they are dense (all experts contribute) or sparse (only a few experts are activated), and how the system handles load balancing and training stability.
Early Routing Designs
One of the earliest routing formulations was the meta-pi network, reported by Hampshire and Waibel in the early 1990s. In this design, the output is the weighted sum of expert outputs, and training proceeds by gradient descent on the mean-squared error loss. The experts could be arbitrary functions, and the gating network learned to assign weights based on the input. In their original publication, the researchers applied this to classifying phonemes in speech signals from six Japanese speakers (two female, four male). They trained six experts, each a time-delayed neural network operating on mel spectrograms. Notably, the learned routing dedicated five experts to five individual speakers, while the sixth male speaker's voice was classified by a linear combination of the experts for the other three male speakers, demonstrating that routing can discover shared subspaces rather than one-to-one mappings.
Another early approach was the adaptive mixtures of local experts, which used a Gaussian mixture model for the gating function. Here, each expert predicted a Gaussian distribution over the output, often ignoring the input entirely and simply learning a mean vector. The weighting function was a linear-softmax function, where the weight for expert \(i\) was computed as \(w(x)_i = \exp(k_i^T x + b_i) / \sum_j \exp(k_j^T x + b_j)\). This softmax routing produced a normalized probability distribution over experts, and the overall model output was a mixture of Gaussian predictions. This formulation allowed for probabilistic interpretation and was trained via maximum likelihood, providing a foundation for later probabilistic routing methods.
Sparse Routing and the Transformer Era
Modern mixture of experts routing gained prominence with the rise of the Transformer (architecture) architecture in Deep learning. In large-scale models, dense routing - where every expert processes every input - becomes computationally prohibitive as the number of experts grows. Sparse routing, introduced in the 2017 paper "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" by researchers including Noam Shazeer and colleagues at Google DeepMind, addressed this by activating only a small subset of experts per input token. The gating function computes a softmax over expert scores but then applies a top-k selection, keeping only the highest-scoring experts (typically k = 1 or 2) and zeroing out the rest. This allows the model to have billions of parameters while only computing a fraction of them for each forward pass.
The sparse gating function typically uses a trainable weight matrix that maps the input token representation to a vector of logits, one per expert. These logits are passed through a softmax to produce probabilities, and the top-k experts are selected. The outputs of the selected experts are weighted by their normalized probabilities and summed. This routing mechanism is differentiable with respect to the gating parameters, allowing end-to-end training via backpropagation. However, sparse routing introduces challenges such as load imbalance, where a few experts dominate and others receive little training signal, leading to the development of auxiliary load-balancing losses.
Load Balancing and Auxiliary Losses
A persistent issue in mixture of experts routing is expert collapse, where the gating network learns to route most inputs to a small set of experts, leaving others underutilized. To counter this, modern implementations add auxiliary losses that encourage uniform routing. One common approach, used in models like the Switch Transformer (introduced by Google DeepMind in 2021), adds a load-balancing loss that penalizes the gating function when the distribution of tokens assigned to experts deviates from uniform. This loss is typically computed as the scaled dot product between the average routing probability and the fraction of tokens routed to each expert, and it is added to the main training loss with a small coefficient. Another technique is the use of expert capacity limits, where each expert can only process a fixed number of tokens per batch; tokens that exceed this capacity are dropped or routed to a residual connection, preventing any single expert from becoming a bottleneck.
More recent methods, such as the DeepSeekMoE architecture, refine load balancing by using fine-grained expert segmentation and shared experts. In this design, experts are split into smaller units, and a small number of shared experts are always activated, while the remaining routed experts are selected via a gating function. This reduces the redundancy in expert specialization and improves parameter efficiency. The routing in such models often employs a sigmoid-based gating rather than softmax, allowing multiple experts to be activated independently, and uses a bias term that is adjusted during training to balance load without interfering with the main loss gradient.
Routing in Large Language Models
Mixture of experts routing has become a cornerstone of scaling Large language models. Models such as Mixtral 8x7B, developed by Mistral AI, use a sparse MoE layer where each token is routed to two of eight experts, each expert being a feed-forward network. This allows the model to have 47 billion total parameters but only use about 13 billion per token, matching the inference cost of a much smaller dense model. Similarly, the Switch Transformer demonstrated that scaling to trillions of parameters is feasible with sparse routing, achieving speedups over dense baselines on natural language tasks. In these models, the routing decision is made per token, not per sequence, allowing the model to allocate different experts to different parts of the input.
The routing mechanism in large language models often operates on the hidden states produced by Multi-Head Attention layers. The gating function is a linear projection followed by a softmax or sigmoid, and it is trained jointly with the rest of the network. One key design choice is whether to use a noisy top-k gating, where trainable Gaussian noise is added to the logits before selection, which encourages exploration during training and prevents the gating from becoming too deterministic early on. Another choice is the use of expert parallelism, where experts are distributed across multiple devices, and routing must be coordinated to minimize communication overhead. This has led to specialized implementations in frameworks like Amazon Web Services SageMaker and Google Cloud TPU environments.
Challenges and Recent Advances
Despite its effectiveness, mixture of experts routing faces several open challenges. One is the trade-off between routing granularity and computational efficiency: too many experts can lead to memory overhead and communication costs, while too few limit capacity. Another is the instability of training, as the discrete top-k selection can cause gradient issues, though straight-through estimators and softmax relaxation have been explored. Recent work has also investigated learned routing policies that adapt over time, such as using reinforcement learning to optimize routing decisions, though these are less common in production systems.
Another area of active research is the interpretability of routing. Studies have shown that experts in large models often specialize in semantic or syntactic categories, such as punctuation, mathematical reasoning, or code, but the mapping is not always clean. Techniques like routing analysis, where the distribution of tokens per expert is visualized, have been used to understand these patterns. Additionally, some models employ a hierarchical routing scheme, where a first-level router selects a group of experts and a second-level router selects within the group, reducing the number of comparisons needed. This is particularly useful when the number of experts is very large, as in the case of models with thousands of experts.
Comparison with Dense Models
Mixture of experts routing offers a fundamental alternative to dense models, where every parameter is used for every input. Dense models, such as the original Transformer (architecture) architectures, have a fixed computational cost per token, while MoE models have a variable cost depending on routing decisions. This allows MoE models to achieve higher capacity without a proportional increase in inference cost, making them attractive for deployment in resource-constrained environments. However, dense models are often easier to train and fine-tune, as they do not require load-balancing losses or careful capacity tuning. The choice between dense and MoE architectures depends on the specific application, with MoE being particularly suited for large-scale training where the goal is to maximize quality given a fixed compute budget.
In practice, many organizations, including OpenAI, Anthropic, and Google DeepMind, have adopted MoE layers in their production models, though they often do not disclose the exact routing details. The technique has also been applied beyond language, including in computer vision and speech recognition, where it has shown similar benefits. As hardware continues to evolve, with specialized accelerators like AWS Trainium and Groq optimizing for sparse computation, the efficiency of MoE routing is likely to improve further, making it a standard component of future Artificial intelligence systems.
Future Directions
The future of mixture of experts routing lies in making it more adaptive and efficient. One direction is the development of fully differentiable routing, where the discrete top-k selection is replaced by a continuous approximation, allowing for smoother gradients and potentially better optimization. Another is the use of learned routing policies that can dynamically adjust the number of active experts based on input complexity, reducing computation for simple inputs and increasing it for complex ones. Additionally, research into expert merging and pruning, where redundant experts are combined or removed after training, could reduce memory footprint without sacrificing quality. As models continue to scale, the routing mechanism will play an increasingly central role in determining their performance and practicality, making it a rich area for both theoretical and applied research.