4. Vectors
A token ID is just an integer. 4122 does not mean anything to a pile of matrix multiplies. The first real move the model makes is to replace each ID with a list of numbers.
That list is called a vector. The list of all such lists — one learned vector per vocabulary entry — is the embedding table. It is a lookup.
token 4122 → [ 0.12, -0.44, 0.08, ... 8192 numbers ... ]
token 653 → [ -0.03, 0.91, 0.17, ... ]Qwen3.8's vectors are 8,192 numbers long. That is the hidden size. Every layer of the model will keep talking in 8,192-dimensional lists. The whole network is a machine for revising those lists.
What is in the 8,192 numbers? Nothing you can read. There is no slot that means "this is a noun" or "this is Python." Early in training the table is random. By the end, tokens that appear in similar places have landed near each other. cat and dog end up closer than cat and photosynthesis. def ends up near other Python. This happens because it helps with next-token prediction, not because anyone labeled the dimensions.
A programmer picture that is almost right: each token gets a fat struct of floats. The rest of the model is a pipeline that updates every struct, over and over, using the other structs as context. At the end, one last matrix turns the last struct back into 248,320 scores, one per possible next token.
each one 8192 floats"] C --> D["layer 1 revises them"] D --> E["layer 2 revises them"] E --> F["..."] F --> G["layer 92 revises them"] G --> H["last vector → 248,320 scores"]
Two more words you will keep seeing.
Residual stream. People call the running list of vectors the residual stream because each layer does not replace them from scratch. It computes a change and adds the change back. Like:
x = x + attention(x)
x = x + feedforward(x)The original signal is still in there. The layer wrote in the margins. This is one of the reasons deep networks train at all. If a layer is useless, adding almost-zero does not destroy what the previous layers already figured out.
Dimension. When someone says "the model is 8192-wide," they mean each token's vector has 8,192 slots. Wider usually means more room to store stuff, and more compute per layer. Deeper — more layers — means more rounds of revision. Qwen3.8 is 92 layers deep and 8192 wide. Those two numbers, plus the expert setup in chapter 9, are most of its shape.
You do not need linear algebra to go further. Hold onto this:
- A token becomes a vector.
- Every layer reads those vectors and writes an update onto them.
- The last vector of the last token is what gets turned into "what comes next."
Attention, in the next chapter, is just the part of a layer that lets one token's vector look at the other tokens' vectors before it updates itself.