26. What labs still hide

You can know a lot from papers. You cannot know the job.

Here is the split, as of this writing, without pretending.

Public enough to learn from

• The shapes of parallelism (Llama 3's 4D writeup, Megatron-Core docs, DeepSeek-V3 §3).
• That MoE all-to-all is the bottleneck, and that custom kernels (DeepEP, Hybrid-EP) are now part of the model.
• That FP8 can work at 671B if you are careful about scaling and what you keep in high precision.
• That attention sinks and massive activations are real, measurable, and that gating plus better norms are the current treatment (Qwen).
• That agent RL is a factory of environments and judges, not a loss function (Qwen3.8 blog, R1/GRPO papers).
• Rough GPU-hour receipts, sometimes (DeepSeek). Rough cluster sizes, sometimes (xAI, Meta).

Still tribal knowledge

• The exact parallel map for Qwen3.8-2.4T. 512 experts, 92 layers, 8192 wide — they have not said TP/PP/EP/DP. You can guess it looks like a bigger Qwen3-Next / DeepSeek hybrid. Guessing is not knowing.
• How they schedule RL against a hybrid recurrent+MoE model without the farm stalling. They said they "solved" it. Not how.
• The true mix of pretrain data, and how they deal with other models' output in the crawl.
• The real MFU of a 100k-GPU Ethernet training job. NVIDIA's 95% network throughput is not MFU.
• How often they roll back. DeepSeek said never, for V3. That is famous because it is rare.
• The judge models, the hidden evals, the canary tasks that tell them the model is hacking.
• Kernel-level stuff that never hits GitHub: fused gated-DeltaNet+MoE, their all-to-all, their checkpoint format.
• Power, cooling, and the operations playbook when a row of racks dies at 3 a.m.

What "reading what xAI reads" actually means

Not a secret blog. A stack like this:

1. DeepSeek-V3 technical report, infrastructure section, twice.
2. DeepSeek-V3 hardware reflections (the H800 constraints paper).
3. Llama 3 training-at-scale parallelism paper.
4. Megatron-Core parallelism guide, then the MoE folding paper.
5. Qwen3 report (data + post-train) and Qwen3-Next (stability + hybrid).
6. vLLM PagedAttention and SGLang's serving notes, because half the org serves.
7. R1 / GRPO, then whatever the lab's current async-RL doc is.
8. Your own traces. Loss. MFU. Expert load histograms. Cache hit rate. Rollback count.

xAI's public surface is Colossus and a few Grok paragraphs. The work is the same work. They have more GPUs and fewer papers. That makes them harder to learn from, not more advanced by default.

The meta-hard problem

Every fix in these chapters is local. Overlap all-to-all and you can add experts. Gate attention and you can quantize. Disaggregate prefill and you can sell 1M context. Stand up a rollout farm and you can RL on repos.

The remaining hard problem is that these fixes fight. The parallel map that is good for pretrain is bad for RL decode. The gate that kills sinks can change the features the judge learned to look at. The data mix that lifts code can drop multilingual. The Ethernet fabric that lets you build a cluster in 122 days can lose a collective in a way IB would not.

So the job at a lab is not "know MoE." It is "keep a 10,000-line configuration, a 100,000-GPU animal, and a model that is still learning, all pointed at the same loss, for long enough that the simple loop from chapter 3 compounds."

That is the un-easy part. It is not a harder idea. It is a harder machine.