3. The only trick

Here is the entire training objective, the thing the model is built to do:

Given the tokens so far, guess the next one.

That is it. Not "understand the question." Not "be helpful." Not "know facts." Guess the next token.

If the text so far is:

The capital of France is

a well-trained model puts a lot of probability on Paris. If the text so far is a half-written Python function, it puts probability on the tokens that would correctly finish the function. If the text so far is a rude user and a polite assistant, it puts probability on a polite reply.

Everything else people claim about language models is a side effect of getting good at this one game.

This is worth sitting with, because it explains a lot of behavior that looks mysterious if you think the model is trying to be a person.

It is autocomplete. A much better autocomplete than your phone, trained on a terrifying amount of text, but autocomplete. Your phone suggests the next word because that word often follows these words. Qwen3.8 does the same thing, at a scale where "often follows" starts to look like reasoning.

It is not retrieving. When it says Paris is the capital of France, it is not looking up a row in a table. The association got baked into the weights because, in the training text, Paris really did keep following The capital of France is. If the training text had been wrong, the model would be wrong in the same way, with the same confidence.

It can only move forward. The model does not jump around in the answer. It emits token 1, then token 2, then token 3. If token 4 needs something it should have said in token 2, it is stuck with what it already wrote. This is why models talk themselves into corners, and why "think first, then answer" helps. The thinking tokens become part of text_so_far for the answer tokens.

It will finish the pattern in front of it. If you write like a confused person, it continues as a confused person. If you write like a careful engineer, it continues as a careful engineer. If the last thing in the prompt is a half-finished lie, a next-token model is under pressure to finish the lie. This is not a moral failing. It is the objective.

Training a model, at the first and largest stage, is just playing this game on a mountain of text. Hide the next token. Make a guess. Measure how wrong the guess was. Nudge every weight a tiny amount so the next guess on a similar example is slightly less wrong. Do that trillions of times.

The measure of wrongness is called loss. You do not need the formula. Lower is better. When people say a training run is "going well," they mean the loss is falling.

A useful picture:

training text:    The cat sat on the mat
the model sees:   The cat sat on the ???
it guesses:       [mat: 0.41, floor: 0.12, couch: 0.09, ...]
actual next:      mat
loss:             "you were not sure enough about mat"
then:             tweak millions of weights a microscopic amount

No one is telling it what a cat is. No one is writing a rule about furniture. If it sees enough sentences, the weights settle into a shape where mat is a good guess after sat on the.

That is pretraining. Later chapters cover the stages that come after — teaching it to follow instructions, to be less of a jerk, to think out loud. Those stages change which next token it prefers. They do not change the fact that next-token prediction is the only move it has.

Once you see this, a lot of product features snap into focus.

A chatbot is a next-token model with a prompt that looks like a conversation.

A "reasoning" model is a next-token model that has been trained to emit a long scratchpad before the answer, because those scratchpad tokens help the later tokens be right.

A coding agent is a next-token model whose text_so_far includes tool results, file contents, and error messages, so the next tokens can be the next command.

Same function. Different text in front of it.