Quantization
Shrinking a model by storing its numbers at lower precision, so it uses less memory and runs faster, with a small quality trade-off.
It is what lets large open-weight models run on ordinary computers instead of only on servers.
Frequently asked questions
How can a big AI model run on an ordinary computer?
Often through quantization, which shrinks the model by storing its numbers at lower precision so it uses less memory and runs faster. That is what lets large open-weight models run on regular machines instead of only on servers.
Does quantization hurt the quality of the AI?
There is usually a small quality trade-off, but for many everyday tasks the difference is minor and well worth the speed and memory savings. How much it matters depends on how aggressively the model is compressed.
Is quantization the same as making a smaller model?
No. Quantization keeps the same model but stores its existing numbers more compactly, while distillation trains a genuinely smaller model to imitate a bigger one. Quantization compresses, distillation rebuilds smaller.
New to all this? Start with what an AI agent really is, browse the full glossary, or explore the learning hub.
← Back to the glossary