Glossary

Short definitions in the same language as the book. If a word only appears once, it may not be here.

Activation checkpointing. Recompute some forward tensors on the backward pass instead of storing them. Spend compute, buy RAM.

Active parameters. How many weights actually run for one token. Qwen3.8: 95B.

All-to-all. The MoE commute: every GPU sends tokens to the GPUs that own the chosen experts, then gets results back.

Attention sink. Softmax has to put its mass somewhere; it often parks it on the first token.

Attention. The step where tokens look at other tokens and mix information.

BPE / tokenizer. The codec that turns text into IDs and back. Qwen3.8's vocabulary is 248,320.

Context / sequence parallelism. Split a long sequence across GPUs. Ship keys and values as needed.

Context window. How many tokens one request can see. Native 262,144; stretched toward ~1M.

Continuous batching. Serving requests join and leave a decode batch every step, instead of waiting for a full batch.

Decode. Generating the answer one token at a time, reusing the cache.

Data parallelism / FSDP / ZeRO. Copy or shard the training state across GPUs; different GPUs see different batches.

Dense model. Every token uses the same full feed-forward network. Opposite of MoE.

Distillation. Training a smaller model to imitate a larger one.

Embedding. The lookup that turns a token ID into a vector.

Expert. One MLP in an MoE layer. Qwen3.8 has 512, plus the routing.

Expert parallelism. Put different experts on different GPUs. Creates all-to-all.

Gated DeltaNet. The cheap attention cousin: a running notebook instead of looking at every past token.

GQA. Grouped query attention. Many query heads share fewer key/value heads, to shrink the cache.

Loss spike. The training loss jumps; sometimes the run is over unless you roll back.

Massive activation. A few hidden dimensions become huge; BF16 and quantization suffer.

GRPO. A reinforcement-learning recipe Qwen3 used: compare several attempts at the same problem and reinforce the better ones.

Hidden size. Length of each token's vector. Qwen3.8: 8,192.

KV cache. Stored keys and values for tokens already seen in this request, so decode does not recompute them.

Logits. Raw next-token scores, before softmax.

Loss. How wrong the next-token guess was. Training is mostly "make this number go down."

MoE / mixture of experts. A feed-forward layer made of many MLPs, of which only a few run per token.

MFU. Model FLOPs utilization. How much of the GPU's theoretical math you actually did. 40–50% is good at scale.

MTP. Multi-token prediction. Training (and sometimes decoding) that guesses more than one token ahead.

PagedAttention. Treat the KV cache like virtual memory: fixed pages, less waste.

Pipeline parallelism. Split the model by depth. Micro-batches walk the stages.

Prefill. The one-shot pass over the prompt that fills the cache.

Pretraining. The first, huge stage: next-token prediction on a mountain of text.

Residual stream. The running vectors that each layer reads and writes an update onto.

Reward hacking. The model maximizes the judge instead of the real goal.

RLHF / preference training. Steering the model using comparisons of better vs worse answers.

Rollout. One full attempt at a task, used as RL training data.

RMSNorm. A simple rescaling that keeps vector size under control.

RoPE. A way to bake position into attention by rotating parts of the query and key.

Router. The tiny network that picks which experts run for this token.

SFT. Supervised fine-tuning. Training on example conversations so the model acts like an assistant.

Speculative decoding. Guess extra tokens cheaply, verify with the big model in one pass.

Shared expert. An expert that always runs, in addition to the routed ones.

Softmax. Turns a list of scores into a list of probabilities that add to 1.

SwiGLU. The gated MLP shape used inside experts / feed-forwards.

Straggler. The one slow GPU that sets the step time for everyone.

Temperature / top-p / top-k. Knobs that reshape the next-token die before you roll it.

Tensor parallelism. Split one matrix multiply across GPUs. Chatty; keep it inside a node.

Thinking / chain of thought. Extra tokens emitted before the answer, used as a scratchpad.

Token. A chunk of text with an ID. The model's alphabet.

Total parameters. How many weights you have to store. Qwen3.8: 2.4T.

Weights. The learned numbers. The actual model.

YaRN. A position-scaling trick used to stretch context past the native trained length.

Zero-centered RMSNorm. Store the norm scale as 1 + δ and decay δ, so it cannot grow without bound.