Why batching starts out nearly free
Decoding one token reads every active weight once and does about two operations with each. The chip spends most of the step waiting for memory. If eight requests share that step, the weights are read once and applied eight times. Total throughput rises almost eightfold, and each user's speed barely changes.
So batching does not so much spend compute as use compute that was sitting idle. It stays nearly free until the arithmetic catches up with the memory reads.
What it actually costs
First, latency: past the crossover, every extra request makes each step longer, so every user gets tokens more slowly. Second, memory: every request needs its own KV cache, and on most deployments that, not the weights, caps concurrency. Third, KV reads: weights are read once per step, but each request's cache must be read separately. With long contexts, those reads dominate and batching stops helping.
- Per user
- 16.0 tok/s
- Total
- 128 tok/s
- Step is limited by
- Weight reads
- KV cache memory
- 10 GB
- Largest batch that meets target
- 10
The per-user target caps the batch at 10. Memory caps it at 161. Latency is the binding limit. Compute becomes the limit at batch 13.
A roofline model of one machine, for decode only. Each step takes whichever is longer: arithmetic, or reading the weights once plus every request's KV cache. Hardware figures are adjustable assumptions. Measure a real throughput-versus-batch curve before committing to a number.Continuous batching and chunked prefill
Early servers waited for a whole batch to finish before starting the next. Continuous batching, introduced by the Orca system, schedules at every step instead: finished requests leave and new ones join immediately. Chunked prefill then splits long prompts into pieces and mixes them with ongoing decode steps. A large document arriving does not stall everyone else's output.
How to tune it
The settings that matter are the maximum number of concurrent requests, the token budget per step and the prefill chunk size, the memory reserved for the KV cache and the maximum context per request, and admission control for requests that do not fit. Interactive and bulk workloads are best served by separate pools with separate targets.
The procedure is simple. Set a per-user target, such as a minimum tokens per second and a maximum time to first token. Sweep batch size on the real machine with realistic prompt lengths. Plot total throughput against per-user speed and choose the largest batch that meets the target. The model above shows the shape of that curve. Only a measurement gives the numbers.
The mixture-of-experts caveat
For mixture-of-experts models, each request in a batch pulls in its own experts, so the bytes read per step grow with batch size until nearly every expert is read. Per-user speed drops much faster than for a dense model, then levels off once the union saturates. The middle of that curve is the worst place to operate: large-batch latency without large-batch throughput.
Frequently asked questions
Does batching trade memory bandwidth for compute?
Largely, yes. At small batch sizes it uses compute that would otherwise sit idle, so it costs little. Past the crossover point, compute becomes the limit and each user slows down.
What usually limits concurrency on a single machine?
Memory for the KV cache. Every request needs its own cache, sized by context length, so long documents reduce how many requests fit.