What is stored
Attention lets each token look back at every earlier token. To do that, every layer projects each token into a key (what this token offers to be matched against) and a value (what it contributes when matched). A later token compares its query against all earlier keys and takes a weighted mix of their values.
Without a cache, generating token 1,000 would mean recomputing keys and values for the 999 before it. The cache stores them once. Per layer, the stored tensor has the shape [sequences × KV heads × positions × head size], one for keys and one for values, usually at 16-bit.
The cache is empty. Run prefill.
- Tokens cached
- 0
- Cells written
- 0
- Same cache, 70B-class model
- 0 B
Append-only, never refreshed
Attention in a generative model is causal: token 5 can see tokens 1 to 4 but never token 6. So token 5's keys and values depend only on what came before, and nothing written later can change them. The cache is never refreshed. Prefill writes the prompt's entries, and each decode step appends one position at every layer.
What does happen is eviction and invalidation. The cache is freed when a request finishes, evicted when memory runs short (and recomputed later), trimmed when a model uses a sliding window, and invalidated from the first changed token onward if earlier text changes. Changing the model version or the adapter also invalidates it, because both change the keys and values.
How big it gets
Per token, the cache holds 2 (keys and values) × layers × KV heads × head size × bytes. Grouped-query attention (GQA) shares each key and value head across several query heads, which shrinks this several-fold. Multi-head latent attention (MLA) goes further and caches one compressed vector per layer. Multiply by context length and concurrent conversations to see why the cache, not the weights, often limits how many people a machine can serve.
- Per token
- 320 KB
- Per conversation
- 10 GB
- 4 conversations
- 40 GB
- Conversations that fit
- 12
Uses 31% of the 128 GB budget.
Layer counts and head sizes come from each model's published configuration. The budget is the memory left over after loading the weights.Prefix caching
If two requests start with the same tokens, their cached keys and values for that stretch are identical. A prefix cache keeps them and reuses them. The shared system prompt, a reference document included in every request, and the earlier turns of a chat all qualify.
The match must be exact from the very first token. One different token invalidates everything after it. The practical rule: put stable content first and variable content, such as dates, names and the question itself, last.
Everything up to the new question matches exactly, so only the question needs prefill.
- Tokens reused
- 12,400
- Tokens to prefill
- 60
- Time to first token
- 30 ms
- Without the cache
- 6.2 s
A privacy note
The cache is derived from the prompt, so it contains the customer's data in another form. A cache hit also returns faster than a miss. If users or tenants share a prefix cache, response timing can reveal that someone else sent the same text. Scope cache keys to the tenant and the adapter, and include the cache in data-retention and deletion policies.
Frequently asked questions
Is the KV cache shared between users?
Each request has its own cache. A prefix cache can reuse identical leading segments across requests. In a multi-tenant system it should be scoped per tenant.
Why does putting the date in the system prompt slow things down?
Prefix matches must be exact from the first token. A value that changes daily at the top of the prompt makes every later token a cache miss.