Understanding Quantization for Large Language Models

What quantization changes

Quantization represents model weights and sometimes activations with fewer bits. Moving from floating-point values to lower-precision integers can reduce storage and memory bandwidth, which may make local inference practical. The trade-off is approximation error and hardware-specific behavior.

Linear mapping and calibration

Linear quantization maps a floating-point range onto discrete values through a scale and, for asymmetric schemes, a zero point. Calibration data helps select ranges that represent the values encountered during inference. Outliers can make a single range inefficient, which motivates per-channel or group-wise approaches.

Post-training quantization and quantization-aware training

• PTQ converts a trained model and is usually the faster path to test.
• QAT simulates quantization during training so weights can adapt, but requires training data and more compute.
• Weight-only quantization mainly reduces model memory; quantized activations can add speed on compatible hardware.

GPTQ, AWQ, GGUF and runtime choices

GPTQ and AWQ are weight-quantization approaches with different optimization assumptions. GGUF is a model file format used by local inference runtimes rather than a single quantization algorithm. The right choice depends on the model architecture, runtime and target CPU/GPU/NPU.

Treat published benchmark tables as starting points. Validate output quality, memory and latency with the prompts and hardware that represent the product.