AI from scratch

11. How they train it

The model you download is not born knowing how to chat. It is the end of a long pipeline. Each stage uses the same next-token machine. What changes is the text you grade it on, and how you grade it.

Qwen has not published a full technical report for 3.8. They did for Qwen3, and the 3.8 blog describes the last stage in some detail. What follows is that story: Qwen3's public recipe, plus what 3.8 officially says it added. Where 3.8 is silent, the book says so.

Pretraining

Start with a randomly initialized pile of weights. Show it the internet, more or less. Hide the next token. Grade the guess. Nudge.

This is most of the compute. For Qwen3 it was about 36 trillion tokens, in three moods:

  1. A long general pass, so the model learns language and a pile of facts.
  2. A pass heavier on math, code, and science, so it gets better at hard sequences.
  3. A pass on long documents, so 32k tokens (and later, more) is not a foreign country.

Nobody is teaching it to be an assistant yet. If the web said something false often enough, the model will say it too. If the web is 30% Python, the model will be good at Python. Pretraining is compression of "what text looks like," not "what a good answer is."

A GPU in this picture is just a machine that is fast at the matrix multiplies from the earlier chapters. Training is those multiplies, plus a backwards pass that figures out which weights to nudge. "A training run" is this loop on a cluster, for weeks or months, burning a shocking amount of electricity.

Scaling laws, in plain language: if you grow the model, the data, and the compute together, the loss usually keeps falling. That is why labs keep making bigger models. It has worked. It is also why a 2.4T MoE exists. They wanted more total parameters without paying dense-2.4T prices on every token.

Supervised fine-tuning

Now you stop feeding it raw web pages and start feeding it examples of the thing you actually want.

user: How do I merge two sorted lists?
assistant: Here's a function...

This is SFT. Same next-token loss. Different distribution. You are teaching the model the shape of being an assistant: helpful, formatted, willing to write code, not willing to continue a web forum fight.

The dataset is small compared to pretraining. Quality matters more than size. A few million good conversations can turn a base model into something you can chat with.

Preference training

SFT teaches imitation. It does not teach taste. The model will imitate a so-so answer if that is what the example looked like.

So you collect pairs: same question, two answers, a human (or a stronger model) says which is better. Then you update the weights so the winner becomes more likely and the loser less.

Classic RLHF did this with a second model, a reward model, that learned to score answers, plus a reinforcement-learning loop that asked the language model to chase that score.

DPO is a simpler cousin that skips the reward model and pushes directly on the winner/loser pairs.

You do not need the acronyms. You need this: after SFT, they keep steering the next-token distribution toward answers people (or a judge) prefer, and away from answers they do not.

Reinforcement learning, the part people mean now

The modern version is less "please the human rater" and more "try it, see if it worked."

Give the model a math problem with a known answer. Let it write a long solution. If the final number is right, reward. If the code passes the tests, reward. If the agent actually closed the GitHub issue, reward.

The model generates several attempts (people call these rollouts). The ones that scored better get their tokens reinforced. The ones that failed get pushed down.

Qwen3 used a method called GRPO for this on math and code. The name is not important. The shape is: no separate critic model, compare attempts for the same question against each other, update.

This is why "reasoning models" got good so fast. You do not have to write a textbook of perfect solutions. You need problems with checkable answers, and enough compute to let the model flail, then keep the flailing that worked.

Qwen3.8's blog is unusually specific about the next version of this. They say they scaled real-world RL across agent harnesses — Claude Code, Codex, their own QwenWork, and others — with three pieces:

  1. Lots of environments, grown along task length, workspace messiness, and which harness is in charge.
  2. One reward system that can grade by running code, by rubrics, and by looking at rendered output.
  3. A balancer so each training batch is not accidentally all easy tasks or all one harness.

That is the official story of why 3.8 is pitched as a coworker, not a chatbot. They trained it in the loop it would be used in. Treat the impressive demos (16 days of autonomous coding, a chip design, a simulated year of running shops) as vendor showcases until someone else reproduces them. The method they describe is the part that matters for understanding the model.

Distillation

A big model is a teacher. A small model is a student.

Instead of making the 27B learn everything the hard way, you can have it imitate the big model's answers, or even its probability distributions. Qwen3 says this strong-to-weak distillation worked better, and cheaper, than running the full RL recipe on the little models.

Two flavors:

If and when Qwen3.8-27B ships, this is the likely reason a 27B can feel closer to the flagship than a 27B has any right to. It is not the same model shrunk. It is a smaller network that grew up on the flagship's homework. Qwen has not confirmed that recipe for 3.8 specifically. They did document it for 3.

What open weights do not include

You get the final numbers. You do not get the 36 trillion tokens, or the internal SFT mix, or the reward system. "Open weights" means you can run the function. It does not mean you can repeat the education.