14. One request

Let us walk through what happens when you type something and hit enter.

Say the prompt is: Write a Python function to merge two sorted linked lists.

1. The prompt becomes IDs

The tokenizer chops the sentence, plus whatever chat wrapper the template adds — roles, special tokens, maybe a system message, and for this checkpoint a request to think.

You now have a list of integers. A few dozen for this sentence. Many more if the template is chatty, or if preserve_thinking stuffed an old scratchpad in.

2. IDs become vectors

Each ID is a row in a 248,320 × 8,192 table. The prompt is now a sequence of 8,192-float vectors.

3. Prefill

The serving engine runs the whole prompt through all 92 layers once.

At a DeltaNet layer, each token updates the running notebook and reads from it. Cheap.

At a full-attention layer, each token looks at every earlier token, with 64 query heads sharing 4 key/value heads. The keys and values get written into the KV cache.

At every layer, after the mixing, the router wakes 10 experts plus the shared one. 95 billion parameters actually run. The other couple of trillion stay on the shelf.

At the end of prefill, two things exist: a cache (and DeltaNet state) for this request, and a vector for the last prompt token.

4. The first new token

That last vector is projected to 248,320 logits. Softmax, temperature 1.0, top-p 0.95, top-k 20. A token is sampled.

On this checkpoint, the first tokens are almost certainly the start of a think block. <think>, a newline, then words that look like planning.

5. Decode

Each new token is a tiny prefill of length 1. Build its query. Attend against the cache, or update the notebook. Route through 11 experts. Sample again. Append.

This is the slow loop. This is what you watch stream in the UI. A long think plus a long answer is this loop running tens of thousands of times.

If tools are involved — only on the hosted Max side, officially — a sampled token sequence will look like a function call. The harness runs the function, pastes the result back into the prompt, and starts another prefill+decode. From the model's point of view there is no "tool module." There is more text.

6. Stopping

Generation stops when the model emits an end-of-sequence token, or when you hit a length cap. Official advice for agent work: let thinking go as far as 262,144 tokens and the visible answer as far as 131,072, inside the million-token window. That is a lot of money if you are on the API, and a lot of time if you are self-hosting.

What you never see

You never see the 8,192-dimensional vectors. You never see which experts fired. You never see the attention weights, unless someone built a debugger. The only thing that crosses the API boundary is tokens in and tokens out.

That is worth remembering when a model says "I looked at line 40 and then decided." It did not look, in the sense of opening a file. A tool may have put line 40 into the prompt. Then the next-token machine, with that line in text_so_far, produced tokens that claim a decision. Sometimes the claim matches a real pattern in the weights. Sometimes it is a story the model has learned that explanations are supposed to sound like.

The request, mechanically, was still: vectors, 92 layers, sample, repeat.