25. The data is the model
People say "we trained on 15 trillion tokens" the way they used to say "we have a million users." The number is real. It is also the least interesting part.
What a trillion tokens is
Qwen3: about 36T tokens, 119 languages. Qwen3-Next: a 15T uniformly sampled subset of that, and they still beat a denser model that cost more GPU-hours. DeepSeek-V3: 14.8T. Chinchilla, years earlier, argued that most labs under-trained their models relative to size. The industry heard "more data" and also heard "better data." Both stuck.
A token is not a fact. Web scrapes are boilerplate, spam, SEO, license-violating books, the same Stack Overflow answer fifty times, and, increasingly, text that came out of last year's model. If you do not filter, you spend GPU-hours teaching the model to be a worse version of itself.
The actual pipeline looks like this:
- Collect (Common Crawl, licensed books, code hosts, PDFs, your own logs).
- Dedup. Aggressively. At line level and at document level. Fuzzy-hash near-duplicates or you will memorize the same news wire 400 times.
- Filter. Language ID, quality classifiers, toxicity, personal data. Qwen3 says they annotated over 30T tokens along educational value, domain, safety, then mixed at the instance level using small proxy models. That sentence is doing a lot of work.
- Mix. How much code vs math vs dialogue vs other languages is a hyperparameter. It is also a political decision about what the model will be good at.
- Pack into sequences. This is where you silently create the long-context training set, or fail to.
Synthetic data is the multiplier. Qwen3 used Qwen2.5-VL to OCR PDFs, then Qwen2.5 to clean the text, plus math and code models to write textbooks and problems. DeepSeek and everyone else do a version of this. It works until it loops: model writes problems, model solves them, model trains on the solutions, model gets more confident and not more right. The check is held-out human exams, not the synthetic loss.
Contamination
If the eval leaked into training, you are measuring memory. Labs now have whole teams for this. They still get caught. When you see a 93 on PaperBench in a vendor table, one of the adult questions is "did the paper, the official impl, or the github issues sit in the crawl?" You usually cannot answer from the outside.
This is why independent evals matter, and why "open weights" without open data is only half a scientific object. You can probe the function. You cannot see what it ate.
The RL data is a different species
Pretrain tokens are static. RL tasks are programs. A unit-test suite. A hidden website. A chip toolchain. A simulated shop. Qwen3.8's claim is that they scaled those, not just web pages.
The failure modes change. Instead of SEO spam you get:
- environments that flake, so the model learns to retry until the universe is lucky
- tasks the model already solved, so the gradient is zero
- tasks the model never solves, so the gradient is noise
- judges the model can charm
Balancing that mix is the "online data balancer." It is curriculum design, implemented as a streaming service.
What remains closed
No frontier lab, including Qwen on 3.8, publishes:
- the URL list
- the exact mix percentages
- the quality model
- the decontamination procedure in a way you could rerun
- the RL task generator
xAI talks about "massive data collection" and expanding verifiable domains. That is a press sentence. The hard part is a warehouse of raters, crawlers, license deals, and synthetic pipelines that cost as much as the GPUs and get talked about less, because there is no nice diagram.
A slightly bitter but accurate slogan: the weights are a lossy zip of a data pipeline you will never see. Architecture is how you zip. Data is what was in the folder.