2. Tokens

The model does not read English.

It does not see words, or letters, or pixels of text. The first thing that happens to your prompt is that a separate program, the tokenizer, chops it into chunks called tokens and replaces each chunk with an ID.

"How do I merge two lists?"
        │
        ▼
["How", " do", " I", " merge", " two", " lists", "?"]
        │
        ▼
[ 4122,  653,  358,  9161,   1378,  10735,  30 ]

Those integers are what the model actually eats.

A token is usually a word, or a piece of a word, or a space plus a word. Common words get their own token. Rare words get split. tokenization might become token + ization. A space matters. "Hello" and " Hello" are different tokens.

This is not a bug. It is a compression scheme.

The algorithm most modern models use is called byte-pair encoding, BPE. You start with bytes. You count which pairs of pieces show up together most often in a huge pile of text. You merge the winner into a new piece. You do that over and over until you have a vocabulary of a few hundred thousand pieces.

Qwen3.8's vocabulary has 248,320 slots. That is unusually large. Qwen3 used about 151,000. A bigger vocabulary means common words and common code fragments take fewer tokens, which means the model can see more actual content in the same context window. It also means the last layer of the model has to score a quarter of a million possibilities every time it wants to emit one token.

Think of the tokenizer as a codec. Encode on the way in, decode on the way out. The model lives entirely in ID-space.

A few consequences fall out of this immediately.

The model is billed, limited, and timed in tokens, not words. A 1 million token context window is not 1 million words. English prose is often around 0.75 words per token. Code is worse, because names and punctuation fragment. A long Python file can eat tokens fast.

The model can be weird around the cuts. If a word is split, the model has to finish the word before it can "mean" it. This is one reason models sometimes stumble on rare names, or on counting letters in a word. They are not looking at letters. They are looking at chunks.

Special tokens are just more IDs. Start-of-message, end-of-message, <think>, image placeholders — these are entries in the same vocabulary. The model does not have a separate channel for them. It predicts them the same way it predicts the.

Spaces and newlines are tokens too. This is why models care about markdown, and why a missing blank line in a prompt can change the answer. You did not change the meaning. You changed the ID sequence.

When people say a prompt is "1,200 tokens," they mean the tokenizer emitted 1,200 IDs. That is the real length of the input. Everything after this chapter — attention, memory, cost, the 1 million token window — is about that sequence of IDs, not about your English.