9. Experts

A dense model runs the same feed-forward network on every token. Simple. Expensive. If you want the model to know more things, you make that network bigger, and then every token pays for the extra knowledge, even the.

A mixture of experts is the obvious cheat, once you see it.

Instead of one fat MLP, you keep a crowd of thinner ones. For each token, a tiny network called the router looks at the token's vector and picks a handful of specialists. Only those run. The rest sit idle.

router(token) → "use experts 17, 88, 201, ..."
output = weighted sum of those experts' outputs

The specialists are the experts. They are just MLPs. Same shape as the feed-forward from chapter 6, smaller. Nobody labels them. Training discovers that some of them get good at code, or at Chinese, or at punctuation. Sometimes the specialization is clean. Sometimes it is a mess. You cannot count on "expert 17 is the Python guy."

Two numbers describe an MoE model, and people mix them up constantly.

Total parameters is how much you have to store. Every expert, even the ones that sleep. This is disk, and RAM, and the reason Qwen3.8 is a multi-terabyte download.

Active parameters is how much work you do for one token. The router, the chosen experts, the attention, the leftover dense bits. This is what you pay in compute per token.

Qwen3.8 is 2.4 trillion total, 95 billion active. About 4% of the pile lights up for any given token. That is how you get a model that is huge in knowledge and merely enormous in cost, instead of impossible.

The official layout:

• 512 experts in the pool
• 10 of them chosen per token (routed)
• 1 extra expert that always runs (shared)
• each expert expands to 2,048 internally, then back down

The shared expert is a default worker. Every token sees it. The routed ten are the specialists. Eleven MLPs run; 501 do not.

Why not pick one expert? Because a token is not one thing. JSON in a Python file is syntax and data and English. Ten votes, plus a shared base, is more flexible than one.

Why not pick all of them? Then you are back to a dense model, with extra overhead.

The router is the fragile part. If it falls in love with ten popular experts and ignores the rest, you paid to store 512 and you are using 10. Training adds a load-balancing pressure: spread the work around. The config.json even has a knob for how hard to push that (router_aux_loss_coef: 0.001). You do not need the formula. You need the idea that unused experts are wasted money, so the training process is told to not waste them.

A few consequences, plainly.

You cannot run this on a single home GPU. Active compute is 95B, which is already big. The memory problem is 2.4T. All 512 experts have to live somewhere the GPU can reach, unless you do heroic offloading that will make decode crawl. The open 27B they promised is the one meant for normal hardware. It is not out as of the day the 2.4T weights landed.

"Open weights" and "you can serve this" are different sentences. The files are on Hugging Face. Serving them is a cluster job. NVIDIA is writing blog posts about GB300 racks. That is the audience.

MoE is why the API can be relatively cheap. You are billed for the work of 95B, roughly, not 2.4T. The company still had to train and store the whole thing.

One naming thing. The download is Qwen3.8-2.4T-A95B. The A means activated. Once you know that, a lot of Qwen names decode themselves. Qwen3-235B-A22B was 235 billion total, 22 billion active. Same idea, earlier generation, different backbone.