Mixture of experts, commonly abbreviated MoE, is a neural network architecture in which a model contains many specialized subnetworks, called experts, but only a small subset of them is activated for any given input. A routing mechanism, typically a small learned network of its own, examines each input, such as a token in a Transformer (architecture) layer, and decides which experts should process it. This sparse activation pattern allows the total number of parameters in a model to grow very large while the amount of computation required to process each token stays much smaller, since most of the model's parameters sit idle for any particular input.
The core idea dates back to early work on adaptive mixtures of local experts in the early 1990s, well before the deep learning era, but it was revived and scaled up substantially with the rise of large Transformer (architecture)-based models. Google researchers demonstrated sparsely-gated mixture-of-experts layers integrated into large language models in the late 2010s, and the approach saw wide adoption as a way to scale Large language model capacity through the early 2020s. Mistral AI's Mixtral models, released in December 2023, were among the most prominent open demonstrations of the approach outside the largest labs, and MoE architectures were also widely believed, though not always officially confirmed, to underlie some of the largest frontier models from major labs, including reported aspects of GPT-4's design.
How it works
In a typical MoE transformer, the standard dense feed-forward layer found in each transformer block is replaced with a set of parallel expert subnetworks, often numbering from eight to several hundred, along with a gating or router network. For each token passing through the layer, the router computes a score for every expert and selects only the top few, commonly one or two, to actually process that token; the outputs of the selected experts are then combined, often weighted by the router's confidence, to produce the layer's output. Because only a fraction of experts are used per token, the active, or "activated," parameter count for any single forward pass can be far smaller than the model's total parameter count, letting a model with hundreds of billions of total parameters run at roughly the computational cost of a much smaller dense model for Inference (AI).
Advantages and trade-offs
The main advantage of MoE is decoupling total model capacity from per-token compute cost, letting a model store more knowledge and specialized capability across its many experts without a proportional increase in the cost of using it. This has made MoE attractive for both training efficiency, since larger effective capacity can be reached under a fixed compute budget, and for Inference (AI) efficiency at serving time. The trade-offs are real, however: MoE models require substantially more total memory to hold all the experts, even though only a few are used per token, which shifts the cost burden toward memory and storage rather than raw compute; routing can be unstable or imbalanced during training, with some experts receiving far more traffic than others unless specific load-balancing techniques are used; and MoE models have historically been more complex to train and serve efficiently across multiple accelerators than equivalently capable dense models.
Adoption
By 2024 and 2025, mixture-of-experts had become a common architectural choice among both open-weight and closed frontier models, appearing in systems including DeepSeek-R1, several Qwen variants, and Grok, generally as a way to reach very large total parameter counts, often in the hundreds of billions, while keeping active compute per token in a much more modest and serviceable range. The technique is frequently discussed alongside Quantization and Knowledge distillation as one of the main levers labs use to manage the cost of serving very large language models at scale.