Where the precision is lost
Only the weights of the linear layers are quantized: the attention projections and the feed-forward or expert matrices. Together they are more than 95% of a model's parameters. Activations, normalization layers and the arithmetic itself stay at 16-bit. Inside the matrix-multiply kernel, each stored code is expanded back to a 16-bit value just before it is used.
So quantization saves memory and memory traffic, not arithmetic. That is why it speeds up token generation, which is limited by bytes read, much more than prompt processing, which is limited by arithmetic.
Why subtract the minimum, and how the scale is set
The stored codes are unsigned integers. At 4 bits there are only sixteen of them, 0 to 15. Quantizing means laying sixteen evenly spaced values across the group's range and snapping each weight to the nearest one.
Subtracting the group's minimum shifts the range so the smallest weight lands on code 0. Dividing by the scale converts the distance above the minimum into a number of steps. The scale is (max − min) ÷ 15: the finest grid that still reaches every weight in the group without clipping any of them. The minimum and the scale are stored alongside the codes, so reconstruction is code × scale + minimum.
Move the slider below and watch the blue ticks. Those ticks are the only values a weight can become.
- Representable values
- 16
- Step between values
- 3.15e-3
- Mean error (ordinary weights)
- 6.7%
- Stored bits per weight
- 4.50
- 70B-parameter model
- 39 GB
| weight | code | stored as | error | error % |
|---|---|---|---|---|
| 0.00250 | 8 | 0.00403 | 0.00153 | 61% |
| -0.00260 | 6 | -0.00228 | 0.00032 | 12% |
| 0.01280 | 11 | 0.01349 | 0.00069 | 5% |
| 0.00210 | 7 | 0.00087 | -0.00123 | 58% |
| -0.01070 | 3 | -0.01174 | -0.00104 | 10% |
| 0.00720 | 9 | 0.00718 | -0.00002 | 0% |
Outliers are the expensive case
Turn on the outlier. One large weight stretches the group's range, the step size grows with it, and the ordinary weights lose most of their resolution. The error on the other fifteen weights rises several-fold, even though the outlier itself is stored exactly.
This is what better methods address. Smaller groups contain the damage. AWQ gives finer treatment to the weights that multiply large activations. GPTQ adjusts the remaining weights to cancel errors already made. The simple minimum-to-maximum grid is close to the best you can do while looking at weights alone. The gains come from also looking at the inputs.
Each output of a layer is a sum of thousands of weight-times-input products. The rounding errors point in random directions and partly cancel, growing roughly with the square root of the count rather than the count itself. That is why 4-bit works at all.
What the rounding does to the output
The model's final layer produces one score per vocabulary entry, called a logit. A softmax turns the logits into next-token probabilities. Quantization error nudges every logit slightly. Most of the time the same token still wins. When two candidates are nearly tied, the order can swap and a different token comes out.
One changed token changes everything after it, because each token is the input to the next. Two outputs can therefore look completely different while being equally good. Text diffs overstate quality loss. Quality is measured statistically, by how far the probability distribution drifts (KL divergence) and by task evaluations.
The model picks earnings. Full precision picked income, so every token generated after this one differs.
- Same top choice as 16-bit
- 90.7%
- Top-5 overlap
- 88.5%
- Distribution drift (KL)
- 0.0415
- Bytes per weight
- 0.562
Choosing a bit width
Eight-bit weights are typically measured as close to lossless. Four-bit is the common trade-off: half the memory of 8-bit, with small losses concentrated in exact recall of rare names, numbers and citations. Below 4 bits, quality falls quickly.
For regulated work, compute figures with deterministic tools, never in the model. Then check that every figure in the final answer matches its source exactly. Copying a number is the kind of close call that quantization affects first, and a mechanical check is cheap.
Frequently asked questions
Does a 4-bit model do its math in 4-bit?
No. With weight-only quantization, codes are expanded to 16-bit inside the kernel and the arithmetic runs at 16-bit. Only storage and memory traffic shrink.
Why does a 4-bit model take 4.5 bits per weight?
Each group of 64 weights also stores a 16-bit scale and a 16-bit offset. That adds 32 bits per 64 weights, or half a bit per weight.
Is a quantized model safe for calculations?
Use deterministic tools for calculations regardless of precision. Then verify that figures copied into the answer match their source.