22. Long context is a tax

A million-token window is a product feature. It is also a bill that comes due twice: once when you train, once when you serve.

Quadratic is not a vibe

Full attention compares every token to every earlier token. Work goes as n². Memory for the KV cache goes as n.

At 4k tokens, nobody cares. At 128k, the cache is the model. At 1M, a naive cache is a joke.

Do the napkin math for one request, Qwen3.8's full-attention layers only.

• 23 full-attention layers (every 4th of 92)
• 4 KV heads, head dim 256
• K and V, 2 bytes each in bf16

Per token, those layers store:

23 × 4 × 256 × 2 × 2  ≈  94,000 bytes

At 1,000,000 tokens that is about 94 GB of KV cache. For one request. Plus the DeltaNet states, plus the weights for 2.4T parameters (even if only 95B run), plus fragmentation.

This is why chapter 10 exists. Most layers do not add to that 94 GB. A DeltaNet state is a fixed-size notebook, not a list that grows with n. Hybrid attention is how you advertise 1M. It is not how you get 1M for free. The 23 full layers still have to look.

GQA (4 KV heads instead of 64) is the other 16× in that napkin. Without it the cache would be about 1.5 TB per request. You would not serve this. You would write a paper about why you cannot.

Training long is worse than serving long

Serving can use a cache and a single request's batch of 1. Training needs activations for the backward pass, and a batch, and sequences that are actually long, not padded junk.

People do not pretrain from step 0 at 1M. Qwen3 pretrained mostly at 4k, then a long-context stage at 32k. DeepSeek-V3 did a two-stage extension, 32k then 128k, and that extension cost them 119k GPU-hours on top of 2.66M for pretrain. Context is a phase.

The tricks have names:

• YaRN / RoPE scaling. Change how positions map to rotations so the model can be pointed at lengths it rarely saw. Qwen3.8's card says native 262,144, extensible to about 1,010,000. The stretch is inference-time scaling, not "we trained on million-token documents."
• Partial RoPE. Only 25% of the head rotates. The other 75% match on content, so they do not break as badly when the angles go off-distribution. That is in the 3.8 config: partial_rotary_factor: 0.25.
• Document packing and masking. You glue short docs into a long sequence so the GPU sees a long n, but you mask attention so doc A cannot look at doc B. Otherwise the model learns to cheat across a pack.
• Context parallelism. Split n across GPUs, as in chapter 18. The tax is shipping K/V.

InfiniPipe and other 2025 systems exist because real data is skewed. A few sequences are huge, most are not. A static pipeline that assumes every micro-batch is 128k will idle. They pack, split, and change pipeline granularity mid-run. This is unglamorous and it is where the time goes.

Prefill vs decode, again, as a tax

Chapter 8 split serving into prefill and decode. Long context makes that split a product architecture.

Prefill a 500k-token PDF and you want a fat batch of GPUs, high compute, you can tolerate a second of latency. Decode the answer one token at a time and you want the KV close, small batches, low latency. If you do both on the same machines, you compromise both.

Frontier serving increasingly disaggregates them. Prefill pool over here. Decode pool over there. Ship the cache (or a compressed cache) across. DeepSeek describes this for V3. Every serious vLLM / SGLang deployment of a long-context MoE is on this path.

Prefix caching is the cousin: if a thousand users share the same 100k-token system prompt or the same repo dump, compute it once. Qwen's API prices cached input far below fresh input ($0.25 vs $2 / MTok on the international Max table). That price is the tax, itemized.

What "1M context" should mean when you read a spec

Ask four questions:

1. Native trained length, or stretched?
2. How many layers are actually full attention?
3. What is the KV cache in GB at that length, for one request?
4. Is the advertised window shared with a 262k thinking budget?

Qwen3.8-Max's studio page: 1,000,000 window, ~991k max input, 131k max output, 262k max chain-of-thought. The thinking tokens live inside the window. A "1M model" that thinks for 200k tokens has 800k left for your files.

Long context is real. It is also the most expensive line on the spec sheet, and the one where marketing and config.json disagree most often.