18. Cutting the model up

If it does not fit on one GPU, you split the work. There are a handful of standard cuts. Labs stack them. NVIDIA's Megatron-Core docs treat this as a product: pick TP, PP, CP, EP, DP, multiply, that is your GPU count.

GPUs = TP × PP × CP × EP × DP

Each letter is a different way to be short of memory or short of time.

Data parallelism

The obvious one. Copy the model. Give each copy a different batch. Average the gradients.

This scales until the model itself does not fit. Then you stop copying the model and start sharding it. ZeRO / FSDP means GPU 3 only stores one slice of the weights, one slice of the gradients, one slice of Adam. When a layer needs the full weight, everyone all-gathers their slice, computes, then throws the extra away.

ZeRO-1 shards only the optimizer. ZeRO-2 adds gradients. ZeRO-3 adds the weights. More sharding, less memory, more chatter. DeepSeek-V3 trained with ZeRO-1 plus other cuts, and skipped tensor parallelism entirely. That is a flex. It means they had squeezed memory some other way.

Tensor parallelism

Split a single matrix multiply across GPUs. GPU 0 owns the left half of W, GPU 1 the right half. They each compute a piece and talk.

This is the cut you use when one layer does not fit, or when you want to shrink activations. It is expensive. The GPUs have to talk inside the layer, not just at the end of a step. You almost always keep TP inside a node, on NVLink, and you almost always turn on sequence parallelism with it so LayerNorm does not hold a full-width activation.

Llama 3's public writeup says that with their batch and cluster, TP = 8 was the sweet spot: one node, no crossing the slow network for this cut.

Pipeline parallelism

Split by depth. GPUs 0–7 own layers 1–12. GPUs 8–15 own 13–24. A micro-batch walks the pipeline.

The failure mode is the bubble: the start and end of the pipe, when some GPUs have nothing to do. You hide the bubble with many micro-batches and clever schedules (1F1B, interleaved 1F1B, DualPipe). You also pay a little to send activations between stages. That send is cheap compared to tensor-parallel chatter. The hard part is keeping every stage busy and not exploding activation memory on the early stages, which have to hold more in-flight micro-batches.

DeepSeek's DualPipe feeds the pipe from both ends at once and overlaps the MoE communication with compute. They say that without that overlap, their expert-parallel communication would have been about 1:1 with compute — half the step spent waiting on the network.

Context / sequence parallelism

Split the sequence. Token 0–4095 on this group, 4096–8191 on that group. Attention is the problem, because token 8000 still needs keys from token 10. So you ship keys and values around, either as a ring or as an all-gather.

This is how you train 128k context without one GPU holding the whole thing. Llama 3 added a fourth dimension, CP, when they went from 8k to 131k, and they replaced data-parallel replicas with context-parallel shards so the per-GPU batch did not collapse.

Expert parallelism

The MoE cut. GPU 0 owns experts 0–7. GPU 1 owns 8–15. After attention, every token has to go to the GPUs that hold its chosen experts, then come back.

This is the subject of the next chapter. It is the cut that makes 2.4T possible, and the cut that makes the network the model.

How people actually combine them

A dense 405B looks like Llama 3: FSDP + TP + PP, plus CP when the sequence gets long.

An MoE 671B looks like DeepSeek-V3: PP 16 × EP 64 × ZeRO-1, no TP.

An MoE with huge attention and huge experts increasingly looks like MoE Parallel Folding: attention uses one mesh (TP × CP × DP), experts use another (ETP × EP × EDP), same pipeline depth. Attention and MoE have different shapes. Forcing them onto one grid wastes GPUs. NVIDIA published this in 2025 and claimed 49% MFU on Mixtral 8×22B.

MFU is model FLOPs utilization: how much of the GPU's theoretical math you actually did. 40–50% is good at this scale. 20% means you are paying for a supercomputer and getting a very expensive waiting room. People who train for a living watch this number the way backend engineers watch p99.

There is no free split. Memory you save, you pay for in bytes on the wire. The art is putting the chatty cuts on NVLink and the rare cuts on InfiniBand or Ethernet, and overlapping the wire with math so the GPU is never just sitting there.