10. Two kinds of attention

Chapter 5's attention is the good stuff. Every token looks at every earlier token. Nothing important has to be compressed away. It is also the expensive stuff. Cost and memory grow with the square of the length. Fine for a paragraph. Ugly for a million tokens.

So Qwen stopped using it everywhere.

Most layers in Qwen3.8 use a different mechanism called Gated DeltaNet. You can think of it as a running summary.

Instead of keeping every past key and value around and staring at the whole list, the layer keeps a fixed-size state. Each new token updates the state a little — write some new information in, let some old information fade — and reads from that state. The work per token stays roughly constant. The past has been squeezed into a notebook of fixed size.

That is the trade. You gain speed and you lose perfect recall. A notebook cannot hold a million tokens in full fidelity. If the thing you need was twenty pages ago and the notebook summarized it away, it is gone.

Qwen's answer, tested in Qwen3-Next and then used in Qwen3.5 and Qwen3.8, is not to pick one. It is to mix them.

Three cheap layers. One expensive layer. Repeat.

Gated DeltaNet → MoE
Gated DeltaNet → MoE
Gated DeltaNet → MoE
Gated Attention  → MoE
   (repeat this block 23 times)

That is the official layout. 92 layers. 69 of them are linear-attention (the notebook). 23 of them are full attention (the look-at-everyone). The config.json says the same thing as a list: linear_attention, linear_attention, linear_attention, full_attention, over and over. full_attention_interval: 4.

Why this ratio? Qwen says they tried pure linear, pure full, and the mix. The mix won on both quality and cost. Linear attention is fast but bad at "what exactly did I see 40,000 tokens ago?" Full attention is the opposite. A periodic full-attention layer lets the model refresh against the real sequence. The cheap layers carry the ball in between.

Two extra details on the expensive layers, because they show up in the spec and they are not decoration.

They are gated. After attention produces a mix, a learned gate can turn pieces of it down. Qwen found this reduces a nasty failure mode where a few tokens (often the first one) suck up almost all the attention. People call that an attention sink. The gate is a volume knob on what attention is allowed to write.

They use partial RoPE. Only the first 25% of the position dimensions get the rotary twist. Qwen says this helps the model stretch to lengths it did not fully train on. The official number is partial_rotary_factor: 0.25. Combined with extra scaling (YaRN, if you have heard that name), this is how a model trained natively to 262k tokens gets advertised out toward a million.

The cheap layers have their own head counts: 128 value heads, 16 query/key heads, head size 128. You do not need to memorize that. You need to know they are a different machine than the 64/4 full-attention heads. Same residual stream. Different way of mixing it.

This is the actual reason a 1 million token window is even discussable. A pure-attention 2.4T model at that length would drown in cache. Most of Qwen3.8's depth does not build that cache. It updates a state.

The remaining full-attention layers still do. That is why grouped query attention, from chapter 8, still matters. The expensive layers are fewer, but they are not gone. They are the model's periodic chance to actually look.