7. Picking the next word
After the last layer, you have one vector for the last token. That vector is the model's entire opinion about what should come next, compressed into 8,192 numbers.
A final matrix turns it into 248,320 scores, one per vocabulary entry. These raw scores are called logits. They are not probabilities yet. They are "how much the model likes this token," on an arbitrary scale.
Then comes softmax, which is just a way to turn a list of scores into a list of probabilities that add to 1. The biggest logit becomes a large probability. The tiny ones become almost zero. Nothing mystical happens here. It is unit conversion.
Now you have a distribution. You have to pick.
Greedy means take the top one, every time. This is boring and sometimes worse, because the model talks itself into a loop. the the the the.
Sampling means roll a weighted die. If Paris has 0.62 and Lyon has 0.11, Paris wins most of the time, Lyon sometimes.
The knobs you see in APIs are ways to reshape that die before you roll it.
Temperature. Divide the logits by a number before softmax. Temperature 0 is greedy. Temperature 1 is the model's honest distribution. Above 1 flattens it — more surprise, more nonsense. Qwen's official recipe for this model is temperature = 1.0. They want the honest die.
Top-p. Keep the smallest set of tokens whose probabilities add up to p, throw the rest away, then sample. Official recipe: 0.95. Ignore the weird 5% tail.
Top-k. Keep only the k best tokens. Official recipe: 20. Even if the tail is fat, only twenty options get a vote.
That is the whole "creativity" story. There is no creativity module. There is a distribution and a way of clipping it.
Then you glue that token onto the prompt and run the whole model again. That is generation. One token per forward pass, over and over, until you hit an end token or a length limit.
This is why long answers are expensive. A 2,000 token reply is roughly 2,000 full trips through the network. Chapter 8 is about the cache that stops that from being 2,000 times as slow as it sounds.
Two more things that will matter later.
The model can be trained to emit a private scratchpad before the answer. Those scratchpad tokens are real tokens. They cost real money. They also become part of text_so_far, which is why they help. The official open Qwen3.8 checkpoint always does this. You cannot turn it off. The hosted Max product can.
The official sampling config that shipped with the weights is:
temperature: 1.0
top_p: 0.95
top_k: 20If you change those a lot and the model gets worse, that is expected. The weights were tuned against this die.