Four limits, nine stages
Every serving decision trades between four hardware limits. Loading is limited by storage and verification. Prefill is limited by arithmetic. Decode is limited by memory bandwidth. Concurrency is limited by memory capacity for the KV cache. Knowing which limit governs a stage tells you which change will actually speed it up.
Step through the stages below. Each is coloured by the limit that governs it.
Published weights are converted to the runtime's format, usually quantized, then checksummed and signed.
- Convert Hugging Face safetensors to the MLX layout.
- Quantize: 16-bit weights become 4- or 8-bit codes plus a scale per group.
- Record a checksum per file and sign the manifest, so you can later prove which bytes answered a prompt.
- Ship it in an offline bundle. Nothing downloads at runtime.
Why unified memory changes the loading step
On a discrete GPU, weights are copied across a bus into the card's own memory. On Apple silicon, CPU and GPU share one pool. MLX memory-maps the weight files, and the GPU reads the same physical pages. There is no second copy and no transfer step, so model size is bounded by total unified memory rather than a separate video-memory budget.
The cost moves elsewhere. The first read of every page still comes from the SSD, so a cold start on a very large model takes minutes, not seconds. A machine that may take over from a failed one should already have the model loaded.
Two phases per request
Prefill runs every prompt token through the model in one parallel pass. It fills the KV cache and produces the first token, so its duration is the wait before anything appears. Decode then produces one token per step, re-reading the active weights each time. Decode speed is therefore roughly memory bandwidth divided by the bytes of weights read per token. That is why quantization, which shrinks those bytes, speeds up decode far more than prefill.
The companion articles take each stage apart: quantization, the KV cache, prefill and decode, mixture-of-experts routing, and batching.
Where auditability fits
Two stages matter most in a security review. Packaging records a checksum for every weight file and signs the manifest, so you can prove which exact model answered a prompt. Admission writes an audit event before any computation starts, and completion writes the record of what was returned. Both are cheap, and neither can be added reliably after the fact.
Frequently asked questions
Why does the first answer after a restart take longer?
Weights are read from the SSD on first use, and GPU kernels compile the first time they run. A warm-up pass at startup moves both costs out of the first real request.
Which stage decides how fast text appears?
Decode. Each output token requires reading the model's active weights from memory, so memory bandwidth and weight size set the speed.