Quantization

What does quantized mean in AI?

In general, to quantize is to map a continuous range of values onto a smaller set of steps, the way a photo with millions of colors can be saved with only 256. In AI, a quantized model is one whose parameters have been converted to a lower-precision number format. When you see a model labeled 8-bit, 4-bit, or "Q4," that label describes its quantization.

How quantization works

The analogy: rounding prices to the nearest dollar instead of the nearest cent. You lose a little detail, but the numbers take less space and are quicker to work with.

The precise version: Hugging Face's documentation explains that weights are typically stored in full precision (fp32), that half precision (fp16 or bf16) is increasingly common for large models, and that quantization methods can go further, to integer formats like int8 or int4. Some methods quantize on the fly when the model loads; others use calibration data to preserve more accuracy at very low bit widths.

Precision, memory, and speed trade-offs

Memory scales with bits per weight. As rough arithmetic for the weights alone, a 7-billion-parameter model needs about 14GB at 16-bit, about 7GB at 8-bit, and about 3.5GB at 4-bit. Real usage is higher, because running a model also needs working memory.

Speed often improves because there is less data to move around, but the gain depends on the hardware and method, and some setups see little change. Quality usually drops a little at 8-bit and more at very low bit widths; how much depends on the model and the task, so test on your own work.

A real example of quantization

A law office wants a private assistant that never sends documents to an outside server. It downloads an open-weights model, but the full 16-bit version does not fit in its workstation's memory. A 4-bit quantized version of the same model does, and runs well enough for summarizing and drafting, with someone checking the output.

Why quantization matters and when to use it

Quantization is what makes running AI locally practical: on laptops, workstations, and phones, for privacy, offline use, or lower cost. It also lowers serving costs in data centers. It is less relevant if you use hosted models through an app or API, where the provider handles these choices. It is also used during fine-tuning: QLoRA combines 4-bit quantization with LoRA to fine-tune large models on a single GPU.

Related terms

• Parameters: the numbers that quantization stores at lower precision.
• Open weights: you can only quantize a model whose weights you have.
• Inference: quantization mainly speeds up and lightens inference.
• Distillation: a different way to shrink a model, by training a smaller one to imitate a larger one.

Common misconceptions about quantization

• A quantized model is a different, smaller model. It is the same model and architecture with lower-precision numbers.
• Quantization always makes a model faster. It reliably cuts memory; speed gains depend on hardware support.
• Lower bits are always fine. Quality can drop noticeably at very low precision, especially on harder reasoning tasks.

Quantization in practice

If you are deciding between running a model yourself or using a hosted one, our tutorial on choosing an AI model covers open vs hosted trade-offs, and the open weights entry explains what you get when you download a model.