Edited by 
Short answer
Quantization stores a model's weights (and sometimes its activations) in fewer bits, for example 4 instead of 16, so the model needs about a quarter of the memory and runs faster. GGUF is the file format used by llama.cpp, Ollama and LM Studio, with "K-quants" such as Q4_K_M; GPTQ and AWQ are 4-bit methods built for GPU inference servers; FP8 and FP4 are hardware number formats on NVIDIA Hopper and Blackwell. For most local use, Q4_K_M or Q5_K_M keeps quality close to the original.
What quantization does
A model is mostly a huge list of numbers, its weights. Trained models usually store them in BF16 or FP16, 16 bits each, so an 8-billion-parameter model takes about 16 GB. Quantization maps those numbers onto a smaller set of values that need fewer bits. At 4 bits per weight the same model takes about 4–5 GB.
That helps in two ways. The model fits on smaller GPUs, and because generating each token means reading every weight from memory, fewer bytes also means faster generation. On data-center GPUs, low-precision formats also let the Tensor Cores do more math per clock.
How 4-bit can work at all
Four bits can only represent 16 different values. The trick is block scaling: weights are split into small blocks (often 32 to 256 values), and each block stores its own scale factor, sometimes with an offset. Each 4-bit code then means "this many steps of this block's scale", so every block covers its own range. The scales add a little overhead, which is why a "4-bit" format often averages 4.25 to 4.85 bits per weight.
weight ≈ block scale × 4-bit code (+ block offset)
See how much precision each bit width keeps with the number format converter, or practise choosing formats in the Bit Budget game.
GGUF and K-quants
GGUF is the model file format of llama.cpp, also used by Ollama, LM Studio and many other local apps. It runs on NVIDIA, AMD and Apple GPUs and on CPUs. Its quantization types are named by bit width and method:
| GGUF type | Average bits per weight | 8B model size | Typical verdict |
|---|---|---|---|
| Q8_0 | 8.5 | 8.5 GB | Practically lossless |
| Q6_K | 6.56 | 6.6 GB | Very close to the original |
| Q5_K_M | 5.69 | 5.7 GB | Excellent balance |
| Q4_K_M | 4.85 | 4.9 GB | The most popular choice |
| IQ4_XS | 4.25 | 4.3 GB | Smaller 4-bit with an importance matrix |
| Q3_K_M | 3.91 | 3.9 GB | Noticeable quality loss |
| Q2_K | about 3.35 | 3.4 GB | Only for fitting very large models |
The "K" in K-quants refers to llama.cpp's super-block scheme, where blocks of weights are grouped with additional scale levels; the "_S", "_M" and "_L" suffixes keep progressively more of the sensitive layers at higher precision. "IQ" types use an importance matrix, calibrated on sample text, to spend bits where they matter most.
GPTQ, AWQ and bitsandbytes
- GPTQ (2022) quantizes weights layer by layer, using a small calibration set to adjust the remaining weights so each layer's output changes as little as possible. It is widely supported by GPU inference servers such as vLLM and Text Generation Inference.
- AWQ (activation-aware weight quantization, 2023) notices that a small fraction of weights matter most, because they multiply large activations, and protects those channels by scaling them before quantizing. It is fast and usually slightly more accurate than GPTQ at 4 bits.
- bitsandbytes NF4, introduced with QLoRA (2023), is a 4-bit "normal float" format that makes fine-tuning large models on a single GPU possible.
All three are weight-only: weights are stored in 4 bits but converted back to 16 bits for the math. Methods such as SmoothQuant go further and quantize activations too (W8A8), so the matrix multiplication itself runs in 8-bit.
FP8, FP4 and hardware formats
On data-center GPUs, the fastest quantization is the one the Tensor Cores compute in directly. FP8 (E4M3 and E5M2) arrived with NVIDIA Hopper and doubles throughput over BF16; most production inference on H100 and H200 runs models in FP8 with negligible quality loss. FP4 arrived with Blackwell, using block-scaled formats: MXFP4 (OCP standard, one scale per 32 values) and NVIDIA's NVFP4 (one FP8 scale per 16 values). OpenAI released gpt-oss with its expert weights natively in MXFP4. Details are in FP16, BF16, FP8 and FP4 explained.
Which format should you use?
| Your situation | Good choice |
|---|---|
| Running models locally with Ollama, LM Studio or llama.cpp | GGUF Q4_K_M or Q5_K_M; Q8_0 if memory allows |
| Running on a Mac | GGUF, or MLX 4-bit and 8-bit models |
| Serving many users from NVIDIA GPUs with vLLM | AWQ or GPTQ 4-bit, or FP8 on Hopper and newer |
| H100, H200 or B200 in production | FP8, or NVFP4 on Blackwell |
| Fine-tuning a big model on one GPU | QLoRA with bitsandbytes NF4 |
How much quality do you lose?
Quality is usually measured with perplexity (how surprised the model is by real text) and with benchmarks. The general pattern is consistent: 8-bit is nearly identical to the original, 5–6 bits lose very little, 4-bit methods lose a small but measurable amount, and quality falls off quickly below about 3.5 bits. Larger models tolerate quantization better than small ones, so a 70B model in 4-bit usually beats a 13B model in 16-bit while using less memory.
Frequently asked questions
Is Q4_K_M good enough?
For chat and most everyday tasks, yes; it is the default many apps download. For code, maths or very long reasoning, Q5_K_M, Q6_K or Q8_0 are safer if they fit.
Does quantization make models faster?
Usually, because token generation is limited by memory bandwidth and smaller weights mean fewer bytes to read. Prompt processing on a GPU may not speed up with weight-only formats, because the math still runs in 16-bit.
What is the difference between GPTQ and AWQ?
Both produce 4-bit weights for GPU inference. GPTQ minimises each layer's output error using second-order information; AWQ protects the weight channels that multiply the largest activations. AWQ is typically a little faster to produce and slightly more accurate.
See exactly how much memory each format needs in the LLM VRAM calculator, then read KV cache explained to understand the other half of the memory bill.
Sources and further reading
- Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022)
- Lin et al., AWQ: Activation-aware Weight Quantization (2023)
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)
- Xiao et al., SmoothQuant (2022)
- llama.cpp and the GGUF format
- OCP Microscaling (MX) Formats Specification
Leave a Reply