6. One layer

A language model is not one clever idea stacked on top of a mess. It is the same small machine, repeated.

One layer does two jobs, in this order:

1. Let tokens talk to each other. That is attention.
2. Let each token think by itself. That is the feed-forward network.

Then it does both again. And again. Qwen3.8 does this 92 times.

x = x + attention(normalize(x))
x = x + feedforward(normalize(x))

That is a layer. Almost the entire model is this, ninety-two times, plus a lookup table at the front and a scoring table at the back.

The + is the residual connection from chapter 4. The layer computes a change and adds it. If the change is useless, the previous work survives.

The normalize is RMSNorm. Vectors that have been added to over and over can get huge or tiny, and then the next multiply misbehaves. RMSNorm rescales a vector so its overall size is well-behaved, without trying to be fancy about the mean. Think of it as keeping the volume from drifting, not as adding meaning.

The feed-forward piece is a small MLP that runs on each token separately. No looking at neighbors. Just:

hidden = silu(x @ W_gate) * (x @ W_up)
out    = hidden @ W_down

This shape is called SwiGLU. The names do not matter. What matters is that it is a fat expansion and a squeeze back down, with a gate in the middle so some features can be turned down. Attention mixed information across the sequence. This step processes what just arrived.

Why do this 92 times instead of once, really hard?

Because each pass is cheap locally and the depth is how the model builds longer-range structure. Early layers tend to clean up local stuff — tokenization artifacts, short phrases. Middle layers do a lot of the actual work. Later layers get the vector ready to be scored as a next token. This is a tendency, not a law. Nobody assigned those jobs. They fell out of training.

A useful picture of one token moving through one layer:

In a dense model, the feed-forward is one MLP shared by every token. In Qwen3.8 it is not. The feed-forward is a crowd of smaller MLPs, and only a few of them run for any given token. That is chapter 9.

In a classic transformer, every attention step is the full look-at-everyone version from chapter 5. In Qwen3.8, three out of every four layers use the cheap running-summary version instead. That is chapter 10.

The skeleton does not change. Tokens talk. Then each token thinks. Add. Repeat.

Once you can see a model as a loop, the spec sheet stops being a wall of jargon. "92 layers, hidden size 8192" means: 92 rounds of this, and every vector in the loop is 8,192 floats long. The rest of the sheet is about how they made those two steps cheaper, so they could make the loop wider and the context longer.