17. Why one GPU dies

The loop from the first half of this book fits on a laptop if the model is small. A 7B dense model in 16-bit is about 14 GB of weights. You can chat with it.

Training is a different animal. Inference only needs the weights. Training needs the weights, the gradients, the optimizer's memory of how each weight has been moving, and a huge pile of activations from the forward pass so the backward pass knows what to do.

Here is the usual accounting, in bytes, for one parameter if you train in mixed precision the way most labs still do:

Sixteen bytes per parameter, before you store a single activation.

A 7B model is already about 112 GB of model state — bigger than one 80 GB H100. A 70B is about 1.1 TB. You have not reserved any room for the residual stream, the attention matrices, or fragmentation.

Qwen3.8 is worse in the obvious way and not better in the way people hope. 2.4 trillion parameters × 16 bytes is about 38 TB of model state if you trained it like a dense model. You shard it, because most of those parameters are experts, and you can put different experts on different GPUs. What you do not get is a 16× discount on Adam. Every expert still has weights, gradients, and optimizer slots, even when it sleeps. Sparsity buys FLOPs. It does not buy a smaller optimizer. DeepSeek-V3 is 671B total / 37B active and still carries optimizer state for the 671B. Same tax, different commutes.

Activations are the other wall. During the forward pass you keep the tensors each layer will need for the backward pass. For attention, that cost grows with sequence length. Train at 8k context and it hurts. Train at 128k and the activations can dwarf the weights. This is why long-context training is its own chapter later, and why people recompute some activations on the backward pass instead of storing them. That trick is activation checkpointing: spend extra compute to buy memory.

A programmer picture:

inference:   weights
training:    weights
             + a gradient for every weight
             + two running averages for every weight (Adam)
             + a tape of almost every activation
             + a workspace for the current matmul

This is why "we trained it on one GPU" is never the story for a frontier model, and why "open weights" does not mean "you can continue pretraining." You can run 95B active if you have a rack and you quantize. You cannot train 2.4T unless you are a lab.

DeepSeek published the cleanest public receipt we have. DeepSeek-V3 is 671B total, 37B active. They trained it on 2,048 H800s. The full run — 14.8 trillion pretrain tokens, a long-context extension, and post-training — was 2.788 million H800-hours. At a $2/hour rental they priced that at about $5.6 million for the official run, not counting all the failed experiments. That is the cheap end of frontier, because they refused tensor parallelism and trained in FP8. Meta's Llama 3 405B used 16,384 H100s for months. xAI's public number is a cluster, Colossus, that went from 100,000 Hopper GPUs to 200,000. They have not published a comparable GPU-hour receipt for Grok.

The rest of this half of the book is what you do when the model does not fit. You cut it up. Every cut has a tax. The tax is communication.