8. Memory

The model has no memory of the conversation in the human sense. What it has is the prompt, plus whatever it has already written in this request.

That sounds like it should make long chats cheap. It does not.

Every new token has to attend to the tokens before it. If you recompute that from scratch each time, generating token 10,000 means looking at 10,000 earlier tokens, then 10,001, then 10,002. The bill grows as a triangle. Unusable.

So serving software keeps a KV cache.

Remember queries, keys, and values. For tokens that already exist, their keys and values will not change. You can compute them once and keep them. When a new token arrives, you only build its query, key, and value, then compare that query against the stored keys.

The cache is the model's working memory for this request. It lives in GPU RAM. When the request ends, it dies, unless the product is doing prefix caching across requests.

Two phases, two cost structures.

Prefill is reading your prompt. Highly parallel. GPUs like it. A long prompt makes this part slow and memory-hungry, but it happens once.

Decode is writing the answer, one token at a time. This is usually limited by memory bandwidth — hauling the cache and the weights in and out — not by raw math. This is why a model can read a book faster than it can write one.

The cache is why grouped query attention exists. Qwen3.8's full-attention layers store only 4 key/value heads, not 64. The cache for those layers is about 16 times smaller than it would be with a key/value head per query head. That is not a quality trick. It is a RAM trick.

It is also why long context is a product feature and an engineering problem at the same time.

Qwen3.8's native trained length is 262,144 tokens. The card says this can be stretched to about 1,010,000 with extra position scaling. The hosted Max API advertises a 1,000,000 token window, with about 991k of that available for input and 131k for output. In thinking mode the input budget shrinks a bit more, because the scratchpad can take up to 262,144 tokens by itself.

A million tokens is a lot of English. It is also a lot of cache. Even with grouped query attention, and even with the cheap attention layers from the next chapters, this is datacenter hardware. You do not run the 2.4T model on a laptop. You run it on a rack, or you call the API.

One more distinction that will save you confusion later.

The context window is not "what the model knows." It is "what this request can see." Training put facts and skills into the weights. The window is the scratch paper for this prompt — the files you pasted, the tool output, the earlier turns, the thinking tokens. When the window fills up, old paper falls off the table. The weights do not forget France. They forget the PDF you uploaded 900,000 tokens ago.