Key Takeaway

MoE (Mixture of Experts) keeps several expert FFNs instead of a single one, and a router selects only a few (top-k) of them per token. Per-token compute uses only a few experts, but every expert’s weights still have to sit in memory, so the memory footprint is large.

This post is part of my AI study notes series as I prepare for the SK hynix AI Hackathon preliminary round (Oct 17–18). In the previous post I covered tokens and context windows; today’s post covers the MoE (Mixture of Experts) structure and the memory load it brings.

What Is MoE? How Does Each Token Use Only Some of the Experts?

One-line answer: Instead of one FFN, MoE keeps several expert FFNs, and a router picks only a subset (top-k) per token to compute.

Dense vs MoE: in Dense a token goes through one whole FFN to the output; in MoE the token goes through a router (gate) and only 2 of the experts E1 to E8, E2 and E6, run (top-k=2) before a weighted sum gives the output

  1. Where the FFN used to sit in a transformer block, and how that spot becomes the MoE layer, is covered in the Day2 transformer block post.
  2. The gate network (router) is a small, learned network that decides which expert(s) a token goes to. A token can be sent to more than one expert (top-k), and Switch Transformers route each token to a single expert.
  3. [Analogy] It’s similar to a general hospital’s front desk (the router) directing a symptom (the token) to just two matching departments (the experts).
  4. When the ST-MoE authors trained in a multilingual setup, there was no clean split of experts by language or topic (“there is no single expert specialized in any given language”, Hugging Face Blog). Instead, encoder experts tend to specialize in shallow concepts such as punctuation or proper nouns.

Dense vs. MoE: Why Do Compute and Total Parameter Count Diverge?

One-line answer: Total parameters are huge, but the active parameters actually used per token are only as much as a handful of experts.

  1. In a dense model, every token passes through every parameter, so parameter count is essentially the same as the compute scale per token.
  2. In an MoE model, you have to distinguish “total parameters” from “active parameters” — one cannot directly compare the number of parameters of sparse and dense models (Hugging Face Blog).
  3. The Hugging Face blog uses Mixtral 8x7B as an example. It has 47B parameters, not 8 x 7B = 56B, because “only the FFN layers are treated as individual experts, and the rest of the model parameters are shared.” With top-2, 14B parameters get used, but since attention operations (among others) are shared, “the actual number of used parameters is 12B” (Hugging Face Blog).
  4. MoE works well in high-throughput setups with many machines, while a dense model is better for low-throughput setups with little VRAM. Sparse models are also more prone to overfitting (“Sparse models are more prone to overfitting”, Hugging Face Blog), which historically made fine-tuning harder.
AspectParameter usage (active parameters)Memory requirement (VRAM)Per-token compute speedFine-tuning / training behavior
DenseAll parameters participate in computeScales with parameter countComputes all parametersBaseline
MoE (Sparse)Only a few selected experts are activeLarge (every expert must reside in memory)Much faster for the same parameter countMore prone to overfitting

What Burden Does MoE Put on Memory Capacity and Bandwidth (HBM)?

One-line answer: Even though only a few experts are computed, every expert’s weights still need to be loaded into memory, so the capacity burden is large.

  1. Every parameter, including experts not used for a given token, must be in memory: “all parameters need to be loaded in RAM, so memory requirements are high” (Hugging Face Blog).

MoE memory concept: HBM/VRAM below holds all expert weights E1 to E8, while GPU compute above runs only E2 and E6 selected for this token, with weight reads using bandwidth (no GB scale)

  1. To keep tokens from piling onto a single expert, a load balancing auxiliary loss is added to “encourage giving all experts equal importance” (Hugging Face Blog). Tokens that exceed expert capacity can pass through to the next layer via residual connections or get dropped, and splitting experts across multiple devices is called expert parallelism.
  2. Because this structure needs much more memory capacity relative to its compute, HBM capacity and bandwidth become especially important — this memory-bound angle is covered in the Day3 memory-bound post.
  3. There’s also research on quantization to reduce MoE’s memory footprint. See 「What Is Quantization? How FP16/INT8/INT4 Differ and Why LLM Memory Shrinks」 for details.

[Exam one-liner] MoE splits the FFN into multiple experts and routes tokens through a router to boost inference efficiency. But even the inactive weights have to be loaded into memory, so memory capacity demands are high (HBM capacity and bandwidth matter), and fine-tuning is more prone to overfitting.

Shall We Test What We’ve Learned With a Quiz?

Q1. Does Mixtral 8x7B use all 47B parameters during computation?

A1. No. Per the Hugging Face blog, with top-2 the actual number of used parameters is 12B, but memory still has to hold all 47B.

Q2. In an MoE, does each language such as Korean or English get its own dedicated expert?

A2. No. In the ST-MoE observation no expert specialized in a language; encoder experts tend to split by shallow concepts such as punctuation or proper nouns, rather than cleanly by language or topic.

Q3. What happens if too many tokens pile onto one expert?

A3. Tokens that exceed the expert’s capacity can pass through to the next layer via residual connections or get dropped. An auxiliary loss is used to encourage a more even distribution and prevent this.