Quantization

AI concepts
About 1 min read

An optimization technique that reduces model size and accelerates inference speed by converting high-precision parameters of an AI model to lower bit widths (such as INT8 or INT4). It is an essential compression method for efficiently running large language models (LLMs) in resource-constrained environments.

Also known as
QuantizationModel Quantization

Detailed explanation

Quantization is the process of converting the weights and activation values of an AI model from high precision, such as FP32 (32-bit floating point), to lower-precision formats like INT8, INT4, or NF4. This reduces the model size by 50–80% or more and resolves memory bandwidth bottlenecks, thereby accelerating inference speed. In the LLM domain, algorithms that minimize error, such as AWQ and GPTQ, have advanced to the point where performance degradation is barely noticeable even with 4-bit quantization. It plays a crucial role in running models with billions of parameters on consumer-grade GPUs or in on-device environments.

Why it matters for tool selection

You must choose an appropriately quantized model that matches the capacity of your hardware's VRAM. For instance, while running a 70B model in its original FP16 state requires approximately 140GB of VRAM, using a 4-bit quantized version lowers this requirement to about 40GB, allowing it to run even in high-performance consumer GPU environments.

What to check

  • Format Compatibility: AWQ/GPTQ formats are advantageous when using GPUs, while GGUF is preferred for CPU/Apple Silicon.
  • Performance Loss (Perplexity): Loss is minimal down to 4-bit, but cognitive degradation can become noticeable at 3-bit and below.
  • Hardware Acceleration: Ensure that the accelerator you use (such as NVIDIA Tensor Cores) supports the corresponding integer operations.

Example

Quantizing the Llama 3 8B model to 4-bit (INT4) reduces the model file size from about 15GB to around 5GB, enabling it to run smoothly even on laptops or mobile devices with limited memory.

Commonly confused terms

Pruning

A method of removing low-importance neurons or connections altogether (whereas quantization only modifies the precision of the values).

Distillation

A method of transferring the knowledge of a large model (Teacher) to a smaller model (Student) through training.