MoE (Mixture of Experts)

Technical terms
About 1 min read

A neural network architecture that selectively activates only a subset of 'expert' subnetworks required to process input tokens out of the model's total parameters, simultaneously securing the massive knowledge capacity of large models and efficient computation speeds.

Also known as
Sparse MoESMoEConditional Computation

Detailed explanation

MoE is a structure based on 'sparse activation', utilizing only a fraction of the total parameters for computation. It divides the feed-forward network (FFN) layers of a neural network into multiple expert subnetworks, and a router dynamically distributes tasks by selecting the most suitable expert in real time based on the characteristics of each token. Through this approach, the model's total parameters (knowledge capacity) can be scaled exponentially while keeping the actual computational cost (FLOPs) and latency during inference at a low level. However, since all parameters—including inactive experts—must reside in memory, it requires a large VRAM capacity proportional to the model size. Recently, it has been adopted as a core architecture for high-performance open-source and commercial LLMs, such as Mixtral 8x7B and DeepSeek-V3.

Why it matters in tool selection

MoE models are characterized by being 'faster than dense models of equivalent performance, and smarter than models of equivalent computational cost.' It is an essential architecture to consider when looking to achieve top-tier inference performance while reducing API costs or inference server operation expenses. In particular, the fewer the active parameters relative to the total parameter count, the higher the cost-effectiveness.

What to look for

  • Number of active parameters relative to total parameters: A key metric that determines actual inference speed
  • VRAM requirements: Even if inference speed is fast, a large total parameter size requires a massive amount of high-end GPUs (such as H100)
  • Routing stability: Whether knowledge is learned evenly without overloading specific experts

Key Example

The Mixtral 8x7B model has a total of 46.7B parameters but only activates about 12.9B parameters per token during inference. This allows it to deliver performance comparable to the 70B Llama 2 while being approximately 6 times faster in inference speed.

Related terms

TransformerLLM