Mixture of experts: how routing picks which weights run

Experts replace one part of each layer

A transformer layer has an attention block and a feed-forward block. In a mixture-of-experts (MoE) model, the feed-forward block is replaced by many smaller feed-forward networks, the experts, plus a router. Attention stays shared. The router is a single matrix that turns the token's vector into one score per expert.

Routing, token by token and layer by layer

Input text

layer 1layer 2layer 3layer 4layer 5layer 6

Router at layer 1 for “indemnify”: a score for every expert, keep the top 2

13.4%

5.2%

1.6%

34.6% → gate 0.55

11.0%

4.3%

1.4%

28.6% → gate 0.45

scores = softmax(h · W_router)        # h = this token's vector at this layer
experts = top-2(scores)
output  = Σ gate_i × expert_i(h)      # gates renormalised to sum to 1

Routing decisions in this text: 54
Experts used per layer (of 8): 8 · 8 · 8 · 7 · 8 · 8

An illustrative router with 8 experts per layer and top-2 selection. Its scores are generated, not learned, but the mechanics match a real MoE: each token, at each layer, picks its own experts from that layer's own set. Production models use far more experts, for example 256 per layer with 8 active.

Per token, per layer

The router's only input is the vector for one token at one layer. Nothing in that computation represents the request, the user or the topic. Each layer also has its own router and its own experts: expert 17 in layer 3 is unrelated to expert 17 in layer 40. A 500-token answer through 60 layers makes 30,000 separate routing decisions.

The vector a router sees has already mixed in earlier tokens through attention, so the same word can route differently in different contexts.

So is there a legal expert?

Not in the sense the name suggests. The Mixtral authors examined routing across academic papers, biology, philosophy and source code and found no obvious assignment of experts by topic. The patterns they did see were syntactic, such as indentation in code and consecutive tokens landing on the same expert. Models with many small experts show more statistical skew by domain: feed in legal text and some experts fire more often. But there is no switch that turns on a legal expert, and no way to load only the legal part of a model. Any token can still route to any expert.

All experts stay in memory

A router's choice means reading the chosen experts' weights from memory, not loading them from disk. Every expert must stay resident, because the next token may choose any of them. MoE therefore saves bandwidth, not capacity. A model with 671 billion parameters and 37 billion active reads about as many bytes per token as a 37-billion-parameter dense model, but it needs memory for all 671 billion.

Batches read the union

When several requests share a decode step, each token picks its own experts, and the step must read every expert that anyone picked. As the batch grows, that union approaches the whole model. The model then behaves like a large dense model, and most of the sparsity advantage is gone.

How many experts a batch actually reads

Model

Requests in the batch8

experts read per layer = N × (1 − (1 − k/N)^B)
                       = 256 × (1 − (1 − 8/256)^8) ≈ 57.4

Experts read per layer: 57 of 256
Share of expert weights read: 22%
Versus one request: 7.2× the bytes
Per request: 90% of solo cost

Each request in a batch picks its own experts. A decode step must read every expert that at least one request picked. This assumes requests choose experts independently, which is the worst case. Similar prompts tend to overlap more.

Frequently asked questions

Does a mixture-of-experts model need less memory?

No. All experts must stay resident. It needs less memory bandwidth per token, which makes single-user generation faster.

Can I keep only the experts for my domain?

Not without changing the model. Routing happens per token and per layer, and domain text still uses most experts. Expert pruning is an active research area and needs task-specific evaluation.

Sources & further reading

Talk with us about your workflow →