21. A warehouse that thinks

Once the model is cut up, you still have to keep tens of thousands of GPUs doing the same dance, every two or three seconds, for months.

This is the part that looks like SRE more than ML.

The step is a distributed transaction

A training step is: read a batch, forward, backward, all-reduce or reduce-scatter the gradients, optimizer step, maybe write a checkpoint. If any GPU is late, every GPU is late. The step time is the straggler.

Stragglers come from everywhere. A hot expert. A long sequence in a packed batch. A NIC retry. Thermal throttling. A noisy neighbor on Ethernet. One bad cable. At 16,384 GPUs, rare events are the schedule.

Labs spend a shocking amount of time on load balance below the model: packing similar-length sequences together, dumping pathological documents, pinning traffic to rails, isolating jobs so two training runs do not share a congested switch.

The network is the computer

Inside a node, GPUs talk over NVLink. That path is fat. Between nodes, you get InfiniBand (the HPC default) or a souped-up Ethernet.

xAI's Colossus is the public Ethernet story. NVIDIA says the first phase was 100,000 Hopper GPUs in Memphis, built in 122 days, training 19 days after the first rack, then doubled toward 200,000. The fabric was Spectrum-X Ethernet, not InfiniBand. NVIDIA claims zero application-level packet loss from flow collisions and about 95% throughput with their congestion control. That is a vendor quote about a vendor fabric. What you can take from it: at that size, someone has to make the network behave like a single machine, and the old "just use IB" answer is no longer the only one.

DeepSeek trained V3 on 2,048 H800s — a much smaller cluster — and still wrote custom IB+NVLink kernels and a two-layer fat-tree alternative in a later hardware paper. They care about hops. Every extra switch is microseconds you will feel 92 layers per token.

Power is the other unpublished boss fight. A 100,000-GPU cluster is a small city's electricity. xAI's Memphis build was in the press for gas turbines as much as for GPUs. This is not a footnote. If the power feed is unstable, your MFU is a fantasy.

Failure is the common case

Disks die. GPUs fall off the bus. A node gets a double-bit memory error. At 100,000 devices, something is broken right now.

So you checkpoint. A naive checkpoint of a 2.4T bf16 model is multiple terabytes, and if you stop the world to write it, you have just donated a percentage of the run to S3. The art is asynchronous, sharded checkpoints: each rank writes its slice, overlapping with the next step, with enough redundancy that a dead rank can be replaced from peers.

Then you need elastic restart. Pull the dead node, remap the shards, keep going. A run that cannot survive a daily hardware fault is not a run.

The public numbers are unromantic.

• Meta, via later citations of the Llama 3 16k-GPU run: a hardware failure about every 2.78 hours.
• ByteDance MegaScale: a multi-week, multi-trillion-token job needed more than 100 repair/recover cycles.
• ByteDance ByteRobust (SOSP 2025): on a platform they say is 200k-GPU class, 38k explicit and 6k implicit failures in three months. Implicit includes 5,506 hangs and 148 NaNs. Manual restarts because someone changed code or data: 17%. They still claim 97% effective training time on a 3-month, 9,600-GPU job, by checkpointing every step at under 1% overhead and treating human error as a designed-in failure mode.
• MegaScale again: one slow rank stalls tens of thousands of GPUs, and MFU decays over the life of a job. The cluster gets tired. Finding out why is a visualization problem, not an nvidia-smi problem.

DeepSeek saying they never rolled back is a numerics claim and a reliability claim. Most clusters cannot keep a job alive that long without replacing machines. The product is the restart path.

What xAI has actually said

Almost nothing about software.

Public Grok 4 text: they used Colossus, they ran RL "at pretraining scale," they claim a 6× improvement in training compute efficiency, they expanded verifiable RL data past math and code. No DualPipe equivalent. No EP size. No MFU. No loss curve.

So when people say "what would an xAI engineer read," the honest answer is: the same DeepSeek / Megatron / Qwen / Llama papers everyone else reads, plus internal dashboards we do not have. The cluster size is not a training algorithm. It is a bet that if you keep MFU from collapsing, more GPUs still win.

The number on the wall

Tokens per second per GPU. Everything else is in service of that, given a loss that is still falling.

If you add experts and tokens/sec drop by more than the loss improves, you lost. If you add context length and the cluster idles in attention, you lost. If you flip to FP8 and have to roll back every six hours, you lost.

This is why the simple model in chapter 3 and the warehouse in this chapter are the same job. The warehouse exists to play next-token as many times as possible without the numbers exploding.