AI glossary
What is Mixture of experts?
Also called MoE ·
A mixture of experts (MoE) is a model design that splits a neural network into many smaller "expert" sub-networks and activates only a few of them for each token, so a model can have a huge total parameter count while using a fraction of it per step.
How it works
In a standard ("dense") transformer, every token passes through every parameter of each feed-forward layer. In a mixture-of-experts model, that feed-forward layer is replaced by many parallel experts plus a small router. For each token the router picks the top few experts (often 1 or 2, sometimes 8 out of hundreds) and only those run.
Why it matters
- More capacity for the same compute. Total parameters set how much a model can store; active parameters set how much each token costs to process. MoE separates the two.
- Model names show it. When a model is described as "816B-A42B", it has about 816 billion parameters in total and about 42 billion active per token.
- Memory is still the catch. Only a few experts run per token, but all of them must be loaded, so an MoE model needs memory for its full size even though it computes like a smaller one.
Example
A model with 64 experts per layer and a router that chooses 2 of them runs roughly 1/32 of its expert parameters for each token, which is why large open-weight MoE models can be fast to serve on hardware with enough memory.
Sources
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, arXiv (Shazeer et al., 2017)
- Mixture of Experts Explained, Hugging Face