Same weights, two shapes
A token is never multiplied directly. Its id selects a row of the embedding table, giving a vector of a few thousand numbers. That vector passes through every layer's weight matrices.
In prefill, the vectors for all T prompt tokens are stacked into a [T × width] matrix and multiplied by each weight matrix once. In decode, a single [1 × width] vector, the token just generated, goes through the same matrices. The weights are identical. What changes is how much arithmetic each byte of weights read has to support.
- Time to first token
- 1.3 s
- Prefill throughput
- 1,562 tok/s
- Decode speed, one user
- 177.8 tok/s
- Weights read per decode step
- 4.19 GB
Reading the roofline
A chip can do a fixed number of operations per second and read a fixed number of bytes per second. Their ratio is the ridge point. A workload doing fewer operations per byte than the ridge waits on memory. One doing more waits on arithmetic.
Decode does about 2 operations per weight read, which is far below the ridge on any current hardware, so it waits on memory. Prefill does about 2 × T operations per weight read, so even a short prompt crosses the ridge. That is why quantization, which cuts bytes, speeds up decode, and why a faster GPU, which adds arithmetic, speeds up prefill.
Why long prompts hurt
Prefill cost grows with prompt length, and the attention part grows with its square: every token is compared against every earlier token. Long documents also slow decode, because each step must read the whole KV cache as well as the weights.
Three techniques help. Prefix caching skips prompt text already processed. Chunked prefill splits a long prompt into pieces interleaved with other users' decode steps, so one large document does not freeze everyone's output. Some serving systems go further and run prefill and decode on separate hardware suited to each.
Frequently asked questions
Does prefill multiply prompt tokens by model weights?
Roughly. Each token id is first turned into a vector by an embedding lookup. Prefill stacks those vectors into a matrix and multiplies it by each weight matrix. Attention then compares tokens with each other using keys and values rather than weights.
Why does quantization barely change the time to first token?
Prefill is limited by arithmetic, and weight-only quantization does not reduce arithmetic. It reduces bytes read, which is what limits decode.