20. When the math breaks

The first half of this book treated the numbers as well-behaved floats. At scale they are not. A surprising amount of "architecture research" in 2025–2026 is really numerics: stop the network from producing a few values so large that BF16 cannot represent the rest.

The format is part of the model

FP32 is the honest 32-bit float. Too fat to train in.

BF16 keeps FP32's range and throws away precision. This is the default language of frontier training. Big numbers survive. Small differences between big numbers do not.

FP8 is the current squeeze. Half the bytes of BF16, more math per second on recent NVIDIA parts. DeepSeek-V3 trained a 671B model in FP8 mixed precision and said it was the first time that had worked at that size. They keep some things (often the master weights, some accumulations, some norms) in higher precision, and they quantize in blocks rather than tensor-wide, so one outlier does not wreck a whole matrix.

The failure mode of low precision is not "the answer is 0.1% worse." It is a loss spike: the loss jumps, the weights take a bad step, and sometimes the run is garbage. DeepSeek's flex was not the dollar figure. It was this sentence: we did not experience any irrecoverable loss spikes or perform any rollbacks.

Most labs cannot say that. They keep snapshots and they roll back.

Attention sinks

Softmax has to sum to 1. A head that has nothing useful to look at still has to put its 1.0 somewhere. Empirically, it parks it on the first token. Streaming papers named this the attention sink.

That is not just a curiosity. A huge pile of attention on token 0 means a huge write into the residual stream at a couple of dimensions. Those dimensions become massive activations — values hundreds or thousands of times larger than their neighbors. Sun et al. measured this directly. BF16 gets coarse at that scale. Quantization dies. Long-context tricks that change RoPE frequencies suddenly blow up, because the sink was quietly being used as a bias.

Qwen's fix, in the gated-attention paper and then in Qwen3-Next / 3.5 / 3.8, is almost rude. After attention, multiply by a learned gate. A head that found nothing can output nothing. They report first-token attention dropping from about 47% to 5%, and the max hidden activation from about 1053 to 94, on the experimental models. The blog says this is why they could train the hybrid + sparse-MoE stack without it falling over.

Norms that grow forever

RMSNorm ends with a learned scale, γ. In Qwen3 they also put RMSNorm on Q and K (QK-Norm) to stabilize attention. Then some of those γs became huge. A scale that wants to be 40 is a smell. It means the network is using the norm as a gain knob, and the knob has no stop.

Qwen3-Next switched to zero-centered RMSNorm: store γ as 1 + δ, initialize δ at 0, and weight-decay δ. The scale is constantly pulled back toward 1. Hugging Face's Qwen3-Next code is literally:

output = self._norm(x.float()) * (1.0 + self.weight.float())

with weight initialized to zeros.

They also normalize MoE router weights at init so the first steps do not randomly crown ten experts and starve the rest. Expert collapse at step 200 is a run-killer. You will not find out at step 200,000 that you have been training 10 experts and storing 502.

What a spike actually is

A few common species, none of them mystical:

• An activation exploded, BF16 rounded it into junk, the gradient was junk, the step was junk.
• The router slammed one expert, that expert's gradient was huge, its weights jumped.
• A bad batch (a long document of garbage, a repeated token, a packing bug) produced a huge loss and everyone stepped on it.
• Two parallel cuts got out of sync and you averaged the wrong gradients. This is a software bug that looks like numerics.

The unglamorous machinery around this is: skip-step logic, gradient clipping, loss spike detectors, automatic rollback to the last good checkpoint, and a human oncall who can tell a real spike from a logging glitch. xAI has not published their recipe. Everyone who has trained past 10k GPUs has one.

If you remember one thing: the architecture papers are often about keeping the numbers in a range the hardware can multiply. Gated attention, QK-Norm, zero-centered norms, FP8 block scaling, auxiliary-loss-free routing — different names, same war.