AI from scratch

23. Serving is another model

Training wants every GPU busy on the same step. Serving wants the opposite: many users, bursty arrivals, some prompts huge, some one-line, everyone waiting on the next token.

The weights are the same. The system is not.

The unit of work is a batch that never sits still

If you take requests one at a time, you waste the GPU. If you wait to fill a big batch, the first user stares at a spinner.

Continuous batching (what vLLM made standard) means: at every decode step, a new request can join and a finished one can leave. The batch is a sliding set of sequences, different lengths, different cache sizes. The kernels have to tolerate that. This is why serving is full of ragged tensors and custom attention.

The cache is a heap allocator

Chapter 8's KV cache, naively, is one big contiguous slab per request, sized for the maximum length. Most requests never get there. You fragment. You OOM with free memory you cannot use.

PagedAttention (vLLM) treats the cache like a virtual memory system. Fixed-size pages of KV. A request owns a list of page IDs, not a giant rectangle. You can pack, you can share pages for a common prefix, you can evict.

RadixAttention (SGLang) goes after the sharing. If two requests start with the same system prompt, or the same chat so far, their prefix KV is the same tree node. You store it once. Qwen's cheap cached-input price is this idea, productized.

For Qwen3.8 you have two caches, really: paged KV for the 23 full-attention layers, and a recurrent state for the 69 DeltaNet layers. The second one does not page the same way. Hybrid models made serving engines grow a second code path. That is why the model card tells you to use a recent SGLang / vLLM / TokenSpeed.

Speculative decoding

Decode is memory-bound. The GPU spends most of its time hauling weights and cache, then does a relatively small matmul for one token.

Speculative decoding guesses several future tokens with something cheap — a small draft model, or a head that was trained to predict more than one token — then checks them in one parallel pass of the big model. If the big model agrees, you emit three tokens for roughly the price of one. If it disagrees, you keep the first mismatch and throw the rest away.

Qwen3.8 was trained with MTP, multi-token prediction. That head is the draft. DeepSeek-V3 did the same. The acceptance rate is the whole game. A draft the big model never agrees with is just extra heat.

Quantization is a quality negotiation

FP8 weights (official for 3.8) cut memory and usually keep quality close, especially if the training recipe already used FP8. 4-bit is how the community will try to shrink this. Every bit you drop is a bet that the massive activations from chapter 20 did not land in a place you just rounded to zero. Gated attention and smaller sinks make quantization possible. They are serving features as much as training features.

MoE does not help you as much as you want here. You can quantize experts, but all 512 still have to live in the cluster's memory. Active 95B is the compute. Resident 2.4T is the footprint. Offloading dead experts to CPU RAM is possible and usually fatal to decode-speed. This is why "open weights" for Max-class models mostly means "other companies can host it," not "you can host it."

Disaggregation

Long prefill and token-by-token decode want different machines. Split them. Ship the cache. DeepSeek wrote this down. Everyone is copying it.

A related serving-only MoE trick: redundant hot experts. Watch which experts fire in production. Duplicate the popular ones. DeepSeek said they rebalance about every ten minutes. The router you trained is not the router you serve, not exactly.

What an engineer actually tunes

Serving a frontier MoE is closer to running a database than to calling model.generate(). The model is the storage engine. The product is the query planner.