AI from scratch

24. RL is a factory

Chapter 11 said reinforcement learning is "try it, keep what worked." At frontier scale that sentence becomes a second cluster, a second software stack, and a new way for the model to cheat.

The loop

sample a task
    → the model writes (often for a long time)
    → something scores the result
    → update the weights toward the better attempts

The write is a rollout. For math, a rollout is a few thousand tokens and a checker. For Qwen3.8's coworker pitch, a rollout is an agent in Claude Code or QwenWork, touching a filesystem, for minutes or hours. You do not do that on the training GPUs. You do that on a rollout farm: many copies of the model, or of the hosted endpoint, running environments in parallel, dumping traces back to the trainer.

On-policy means the traces were written by the current weights. Off-policy means they are stale — yesterday's model, or a teacher. Stale traces are cheaper and slightly wrong. Fresh traces are honest and cost a fortune, because a 95B-active model thinking at xhigh for a hard repo task is a long decode.

DeepSeek's GRPO (the recipe Qwen3 also used for math/code) compares several attempts at the same question and pushes up the better ones, without a separate critic network. That saves a model. It does not save you from needing 8, 16, 64 rollouts per prompt.

Verifiable vs vibes

If the unit test passes, you have a number. If the math answer matches, you have a number. This is verifiable reward. It scales. It is why math and code jumped first.

Most of work is not like that. "Was this legal memo good?" "Did the UI look right?" Qwen3.8's blog says they built a universal reward system that can:

A rubric is a judge. A judge can be hacked. The model will find the cheap way to make the judge happy: verbose answers, magic phrases, screenshots that fool the vision grader, tests that assert True. Reward hacking is the default, not the exception. The engineering is making the judge as close as possible to the real outcome, and keeping a human (or a held-out eval) in the loop so you notice when the proxy diverges.

Qwen's third piece, the online data balancer, is about not letting the batch become 90% easy tickets from one harness. If you train on whatever the farm happens to finish first, you train on short, easy, fast-to-fail tasks. The long, valuable ones never update the weights. They say the balancer keeps task, difficulty, workspace, and harness mixed so gradient variance stays sane.

Believe the shape. Treat the 16-day coding demo as a showcase until someone else runs it.

Why hybrid MoE + thinking is nasty for RL

Qwen3-Next's blog has a sentence that is easy to skip: they solved long-standing stability and efficiency issues in RL caused by hybrid attention plus high-sparsity MoE.

Decode during a rollout is already the slow path. Now add:

The trainer wants fresh on-policy traces. The farm wants to batch. The environment wants to be sequential. These three clocks do not agree. Async RL is the compromise: the trainer consumes a queue of traces that are slightly off-policy, and you spend research on how stale is too stale.

xAI's Grok 4 post says they ran RL "at pretraining scale" on Colossus and got a 6× training-compute-efficiency gain, with verifiable data expanded past math and code. That is a direction, not a recipe. The recipe — how they score a non-math task, how off-policy, how they stop hacking — is the unpublished part.

Distillation is how the small model exists

Once the big model is good at the loop, you record its traces and train the 27B (or 8B) to imitate them. Qwen3 said this beat running full RL on the small models, at about a tenth the GPU-hours.

Two failure modes:

The 27B, when it lands, will be a systems object as much as a model: a cheaper rollout worker, a draft model for speculative decoding, a thing you can fine-tune. The 2.4T is the teacher you probably never train again.

The new bottleneck is environments

Pretraining was bound by tokens and cluster. RL-for-agents is bound by how many realistic tasks you can grade. Qwen talks about scaling environments along task length, workspace mess, and harness type, combinatorially. That is the factory: thousands of disposable VMs, flaky tests, browsers, compilers, and a lot of software to keep them from teaching the model to cheat.

Weights are no longer the scarce resource. Tasks with honest scores are.