19. All-to-all

Mixture of experts looked cheap in chapter 9. Eleven MLPs run, 501 sleep. The lie is the commute.

After attention, every token on every GPU has a list of expert IDs. Those experts live on other GPUs. So you dispatch: ship the token's vector to the machines that own those experts. They run the MLP. Then you combine: ship the results back and mix them.

That pattern is all-to-all. Not "broadcast one thing." Every rank has a different, ragged pile of tokens for every other rank. The sizes change every step, because the router just voted.

Standard collectives (NCCL all-to-all) were built for fixed, dense tensors. MoE is sparse and rude. DeepSeek wrote a whole library, DeepEP, because the generic path was eating the step. NVIDIA has Hybrid-EP. Moonshot published pieces of a comms library with Kimi. This is not a research side quest. For a fine-grained MoE, communication is the step.

DeepSeek's numbers, from the V3 report:

• Cross-node expert parallelism made compute:communicate about 1:1 before they overlapped it.
• NVLink inside a node was 160 GB/s, InfiniBand between nodes 50 GB/s, about 3.2× apart.
• They limited each token to at most 4 nodes, so the slow hop stays bounded.
• A token goes IB to the same GPU index on the target node, then NVLink to the GPU that actually holds the expert. The two hops overlap.
• They reserved about 20 SMs (streaming multiprocessors — slices of the GPU) just to drive the network, so the rest could keep multiplying.

That last point is easy to miss. The GPU is not a math box with a network card on the side. If communication kernels hog too many SMs, you have bought an H100 and turned it into a NIC. The current sport is "all-to-all with as close to zero SMs as possible." DeepEP V2 brags about dropping from ~24 SMs to 4–6 for the same bandwidth.

Why the router can ruin your day

If expert 17 becomes popular, GPU 17 gets slammed and everyone else waits. That is load imbalance. The step is as slow as the busiest expert.

Old MoE papers added an auxiliary loss that punishes popular experts. It works. It also, if you crank it, makes the model worse, because you are no longer letting the router be smart — you are making it fair. DeepSeek-V3's trick was auxiliary-loss-free balancing: a bias per expert that they bump up or down based on recent load, used only for the top-k decision, not for the actual mix weights. Qwen3 used a global-batch load-balancing loss so the pressure is computed across the whole batch, not per GPU's myopic view.

There is also token dropping. If an expert is over capacity, throw tokens away (treat them as if they skipped that expert). Training gets faster and noisier. Dropless is cleaner and meaner to the tail latency. Serving often does a cousin of this: redundant experts. DeepSeek, at inference, copies hot experts onto extra GPUs and rebalances every ten minutes from live stats.

Why Qwen3.8's 512 experts is a systems statement

Qwen3 had 128 experts, 8 active. Qwen3-Next and Qwen3.8 went to 512, 10 routed + 1 shared. Qwen says that with global load balancing, more total experts at fixed active count steadily lowers loss. Capacity goes up. The all-to-all fanout goes up with it.

Moonshot's public MoonEP library treats imbalance as a comm problem, not just a loss problem: clone a hot expert for this step so every rank still receives the same number of tokens, keep the tensor shapes static, do not OOM when the router has a tantrum. DeepSeek, at inference, does the cousin — extra copies of hot experts, reshuffled about every ten minutes.

Kimi K2 is the useful counterexample to DualPipe-as-religion. At 1T they rejected DualPipe because it keeps two copies of the weights and gradients, which would have forced more pipeline stages (more bubbles) or more expert-parallel ranks (more all-to-all). They used a smaller EP group and hid comm with extra 1F1B warmup instead. The overlap trick is mandatory. The brand name on the trick is not.

That is the deal you are looking at when you see 2.4T-A95B. The extra trillions are mostly extra specialists. Every token still only wakes eleven of them. Every token may still have to visit several machines to find those eleven.

A useful sentence to steal: MoE moves the bottleneck from FLOPs to the network. If your cluster's intra-node fabric is fat and your inter-node fabric is not, you design the router so tokens do not spray across the building. DeepSeek made that an architectural constraint (node-limited routing). Qwen has not published the equivalent for 3.8. Assume they have one. You cannot ship 512 experts without one.