5. Attention

You are predicting the next token. The useful information is not only in the last word. It is somewhere in the sequence.

In The trophy doesn't fit in the suitcase because it is too big, the word it refers to the trophy. A model that is about to continue this sentence needs the vector for it to have picked up "trophy," not "suitcase."

Attention is how that happens.

Each token, at each layer, gets to look at the other tokens and pull information from the ones that seem relevant. "Seem relevant" is not a rule someone wrote. It is a comparison the model learns to make.

The usual explanation uses three names: query, key, and value. They sound like a database because they are a database, sort of.

For each token, the model builds three vectors from its current vector:

• The query is what this token is looking for.
• The key is what this token looks like to others.
• The value is what this token will hand over if someone picks it.

Then, for token i:

1. Compare query_i to every key_j in the sequence so far.
2. Turn those comparison scores into weights that sum to 1.
3. Build a blend of the value_js using those weights.
4. Add that blend back onto token i.

If it is token 11, its query might match the key for trophy hard and the key for suitcase less. The blend it adds to itself is then mostly trophy-ish. Later layers can use that.

A few details matter in practice.

It only looks backward. When the model is generating, token 20 cannot see token 21. Token 21 does not exist yet. During training they mask the future so the model cannot cheat by reading the answer.

It does this in parallel heads. One head might get good at matching names to pronouns. Another might track indentation. Another might watch quotes. Qwen3.8's full-attention layers have 64 query heads and only 4 key/value heads. That last part is grouped query attention: many queries share a smaller set of keys and values. It is a discount. You keep most of the benefit of many heads, and you store a lot less in memory. Chapter 8 is about why that memory matters.

The comparison is not magic. It is a dot product — "how aligned are these two lists of numbers?" — plus a scaling factor so the numbers do not explode. The weights that sum to 1 come from a softmax, which is a function that turns a list of scores into a list of positive numbers that add to one. Big score becomes almost 1. Small score becomes almost 0.

Positions have to be injected. Attention as described does not know order. dog bites man and man bites dog would look the same if you only had the token vectors. Models add position information so that "near me" and "before me" are visible. Qwen uses RoPE — rotary position embeddings — which rotate parts of the query and key based on position, so relative distance shows up in the comparison. You can think of it as twisting the vectors a little more for tokens that are further apart.

That is standard attention. It is powerful and it is expensive. Every token looks at every earlier token, so the work grows with the square of the length. A 1 million token context with only this kind of attention would be a monster.

Qwen3.8 does not use only this kind. Most of its layers use a cheaper cousin that keeps a running summary instead of staring at the whole past. That is chapter 10. You need this chapter first, because the cheap cousin exists to approximate this.