Quantization is a technique for reducing the numerical precision used to represent a neural network's weights and, in some cases, its activations, in order to shrink model size and reduce the computation required to run it. Neural networks are typically trained using 32-bit or 16-bit floating-point numbers, but many of the values a trained model actually needs to represent well can be approximated using much lower-precision formats, such as 8-bit or even 4-bit integers, with only a small, often acceptable, loss of accuracy. Applied to a Large language model, quantization can reduce a model's memory footprint by a factor of four or more compared to its original training precision, making it possible to run models on hardware with far less memory than would otherwise be required.
Quantization as a general technique in digital signal processing and computing long predates deep learning, but its application to neural networks, and specifically to large language models, became widely practiced starting in the early 2020s as model sizes grew into the tens and hundreds of billions of parameters, making full-precision deployment prohibitively expensive for many users. The release of large open-weight models such as Llama in 2023 drove a surge of community interest in quantization specifically, since it allowed individuals to run models with tens of billions of parameters on a single consumer GPU or even a laptop, something otherwise infeasible at full precision.
How it works
Quantization maps a continuous or high-precision range of weight values onto a smaller, discrete set of lower-precision values, typically by scaling the original values into the target format's representable range and rounding. Post-training quantization applies this conversion to an already-trained model without further training, and is the most common and cheapest approach; it can be as simple as converting every weight independently or more sophisticated, using calibration data to choose scaling factors that minimize the error introduced for the specific model and, sometimes, per layer or per group of weights. Quantization-aware training instead incorporates the effects of reduced precision during the training process itself, which generally preserves more accuracy at very low bit-widths but requires access to training infrastructure and data that most users of a pretrained model do not have. Common target formats as of the mid-2020s included 8-bit integer quantization, which usually produces negligible quality loss, and more aggressive 4-bit and sometimes lower formats, which trade a larger, though often still modest, quality reduction for substantially smaller model size.
Applications
Quantization is one of the primary techniques, alongside Knowledge distillation and architectural approaches such as Mixture of experts, used to make large models practical to deploy outside of data centers with abundant GPU memory. It underlies most local and Edge AI deployment of language models, letting open-weight models run on consumer hardware, single accelerator cards, or even mobile devices. It is also used in production serving infrastructure at large AI labs to reduce the cost of Inference (AI) at scale, since lower-precision computation is generally faster and cheaper on modern accelerators such as GPU (in AI) hardware built with dedicated low-precision arithmetic units. Popular open-source tooling for quantizing and running large models locally, including formats and libraries built around efficient integer and low-bit inference, became widespread parts of the open-weights ecosystem alongside repositories such as Hugging Face.
Trade-offs and limits
The central trade-off in quantization is the balance between size and speed gains versus accuracy loss, and this loss is not uniform: some tasks, particularly those requiring precise arithmetic or fine-grained reasoning, tend to degrade more noticeably under aggressive quantization than open-ended text generation does. Extremely low bit-widths, such as 2-bit or lower, generally produce a substantial and often unacceptable quality drop for most use cases without specialized techniques to compensate. Because quantization is typically applied after training and is comparatively cheap and fast to perform, it is often the first and easiest lever practitioners reach for when a model needs to run under tighter memory or latency constraints than the original full-precision version allows, frequently used together with LoRA-based Fine-tuning techniques such as QLoRA, which combine quantized base weights with a small set of trainable low-rank adapters.