Does quantization affect response speed?

Yes, and in a positive direction. Lower-precision weights mean fewer bytes to transfer from memory per token, so quantized models often generate tokens faster than their FP16 counterparts. A 70B model at Q4_K_M can generate 20-40% more tokens per second than at FP16 on the same hardware, assuming the FP16 version even fits in memory. The speed benefit diminishes at higher quantization levels.

Can I quantize a model myself?

Yes. The llama.cpp project includes a quantize tool that converts FP16 GGUF files to any supported quantization level. Download the FP16 GGUF from Hugging Face, then run the quantize binary with your target format. However, most popular models already have pre-quantized versions available, so manual quantization is only necessary for new or obscure models.

Is Q4_K_M always the best default choice?

For most users and most models, yes. It offers the best balance of quality preservation, memory efficiency, and ecosystem support. The main exceptions are: when you have ample VRAM and want to maximize quality (use Q5_K_M or Q6_K), when you need to squeeze a much larger model into limited VRAM (use Q3_K_M), or when you are on Blackwell hardware (try NVFP4).

Do different models respond differently to quantization?

Yes. Models with more parameters tend to be more resilient to quantization because they have more redundancy in their weights. A 70B model at Q4_K_M loses proportionally less quality than a 7B model at Q4_K_M. Models with mixture-of-experts architecture also tend to handle quantization well because only a subset of parameters is active for any given token. The training methodology matters too: models trained with quantization-aware techniques degrade less gracefully at lower bit widths.

What is the difference between Q4_K_S and Q4_K_M?

Both target approximately 4 bits per weight, but Q4_K_M (Medium) assigns higher precision to more sensitive layers than Q4_K_S (Small). Q4_K_M is about 5-10% larger but preserves slightly more quality. In practice, the difference is small enough that most users default to Q4_K_M for the marginal quality benefit. Q4_K_S is useful when you are right at the edge of fitting and need to save every possible megabyte.

GGUF Quantization Explained: Q4, Q5, Q6, Q8 Compared

CanItRunJune 28, 202610 min readTutorials

What Quantization Is and Why It Matters

Quantization is the process of reducing the numerical precision of a model's weights from their training precision (typically FP32 or BF16) to a lower-bit representation. A model trained in BF16 stores each weight as a 16-bit floating-point number. Quantizing to 4-bit means each weight is stored in roughly 4 bits instead of 16, reducing the model's memory footprint by approximately 4x. This is not merely a storage optimization: it directly determines whether a model fits in your GPU's VRAM and therefore whether you can run it at all. Without quantization, Llama 3.3 70B requires about 140 GB of memory. At Q4_K_M quantization, it needs about 40 GB. That is the difference between needing a cluster of datacenter GPUs and running on a single high-end consumer device. The key question is always how much quality is lost in the process. Modern quantization methods are remarkably good at preserving model behavior. The loss at Q4_K_M is typically 1-3% on academic benchmarks, and often imperceptible in everyday use. At Q2_K, the losses become significant and noticeable. Understanding the quantization spectrum helps you choose the right trade-off for your hardware and tolerance.

The GGUF Format and K-Quants

GGUF (GPT-Generated Unified Format) is the standard file format used by llama.cpp and, by extension, Ollama and most local inference tools. It superseded the older GGML format in 2023 and has become the de facto standard for distributing quantized models. GGUF files contain the quantized weights, model metadata (architecture, vocabulary, hyperparameters), and tokenizer data in a single self-contained file. The K-quant variants (Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, Q8_0) use a mixed-precision approach developed by the llama.cpp project. Instead of quantizing every layer to the same precision, K-quants assign different precision levels to different parts of the model based on their sensitivity. The attention layers, which are most critical for output quality, receive higher precision, while less sensitive feed-forward layers are quantized more aggressively. The S, M, and L suffixes indicate Small, Medium, and Large variants: Q4_K_S uses slightly less memory than Q4_K_M but with marginally lower quality, because it quantizes more layers to lower precision. Q4_K_M is the most commonly used variant because it hits the optimal point on the quality-size curve for 4-bit quantization.

Quantization Levels Compared

Starting from the bottom: Q2_K uses approximately 2.63 bits per weight. It produces the smallest files (roughly 3 GB for a 7B model) but quality degrades significantly. Expect noticeable grammatical errors, factual mistakes, and reduced coherence. Use it only as a last resort when nothing else fits. Q3_K_M uses approximately 3.44 bits per weight. Quality is noticeably below FP16 but still usable for many tasks. This is the lowest level most users should consider for regular use. It is particularly useful for squeezing a larger model class into limited VRAM (e.g., a 27B model on a 16 GB card). Q4_K_M is the default recommendation at approximately 4.50 bits per weight. Quality loss versus FP16 is minimal and often undetectable in casual use. Academic benchmarks typically show a 1-3% drop. This is the sweet spot for most users. Q5_K_M at approximately 5.15 bits per weight offers a small but measurable quality improvement over Q4_K_M for about 15% more memory. Worth using when you have the VRAM headroom. Q6_K at approximately 6.56 bits per weight is very close to FP16 quality. The marginal improvement over Q5_K_M is small. Use it when you have generous VRAM and want to minimize any possible quality loss. Q8_0 at 8 bits per weight is effectively lossless for most practical purposes. Benchmark differences from FP16 are negligible. It uses twice the memory of Q4_K_M, so it is a luxury for models that fit comfortably in your VRAM at this level.

FP16 and BF16: The Baselines

FP16 (half-precision floating point) and BF16 (brain floating point) both use 16 bits per weight but with different numerical ranges. BF16 has a wider exponent range, making it more robust for training, while FP16 has higher precision within its range. For inference, the practical difference is negligible: model outputs are virtually identical between FP16 and BF16. At 16 bits per weight, these formats serve as the quality baseline against which all quantized versions are compared. Running a model at FP16 gives you the full quality of the released weights with no compression artifacts. The cost is memory: a 7B model at FP16 needs about 14 GB, a 70B model needs about 140 GB. Very few consumer setups can run anything beyond small models at FP16. Unless you are doing quality-sensitive research where even the 1% Q4_K_M degradation matters, you should always use a quantized version. The inference speed difference between FP16 and Q4_K_M is also worth noting: Q4_K_M is often faster because smaller weights mean fewer bytes to transfer from memory, and memory bandwidth is the bottleneck for inference.

NVFP4: NVIDIA's Hardware-Accelerated 4-bit

NVFP4 is NVIDIA's native 4-bit floating-point format, supported in hardware on Blackwell architecture GPUs (RTX 5000 series). Unlike GGUF's integer-based Q4_K_M quantization, NVFP4 uses a true floating-point representation at 4 bits, which can preserve more quality for the same bit-width. The Blackwell GPU's tensor cores natively accelerate NVFP4 operations, meaning inference can be both more memory-efficient and faster than Q4_K_M on supported hardware. The catch is that NVFP4 requires a Blackwell GPU: it is not a format you can use on older NVIDIA cards, AMD GPUs, or Apple Silicon. For users with RTX 5090 or RTX 5080 cards, NVFP4 is worth trying as it may offer better quality than Q4_K_M at similar memory usage. However, the llama.cpp ecosystem's support for NVFP4 is still maturing, so Q4_K_M remains the more broadly compatible and well-tested option. If you are buying a new GPU and plan to invest in the NVIDIA ecosystem, NVFP4 support is a meaningful differentiator for the Blackwell generation.

Real Quality Impact by Quantization Level

Abstract percentages are less useful than concrete examples of how quantization affects output. At Q4_K_M, a model like Llama 3.3 70B produces responses that most users cannot distinguish from FP16 in blind tests. The model follows instructions accurately, maintains coherent long-form output, and gets factual questions right at the same rate. Where Q4_K_M occasionally falls short is on edge cases: very precise numerical reasoning, obscure factual knowledge, and tasks that depend on subtle probability distinctions in the model's output distribution. At Q3_K_M, you will start to notice occasional quality issues. The model might misunderstand a nuanced instruction, produce slightly less polished prose, or make factual errors it would not make at higher precision. These errors are not constant and many sessions will be fine, but the error rate increases enough to be noticeable over extended use. At Q2_K, the quality loss is unmistakable. Outputs are often grammatically correct but less coherent. The model is more prone to repetition, non-sequiturs, and confident wrong answers. Use Q2_K only for testing whether a model architecture works for your use case before investing in hardware that can run it at Q4_K_M or better.

Practical Guide: Choosing the Right Quantization

The decision process is straightforward. First, determine your VRAM budget: your GPU's total VRAM minus 500 MB for overhead minus the KV cache memory needed for your target context length. Second, find the highest quantization level where the model weights fit in that budget. Use this rule of thumb: multiply the model's parameter count by the bits-per-weight and divide by 8 to get the approximate file size in GB. A 70B model at Q4_K_M (4.5 bits per weight) uses roughly 70 times 4.5 divided by 8, which is approximately 39 GB. If you are choosing between a larger model at lower quant and a smaller model at higher quant, the larger model usually wins up to Q3_K_M. Below Q3_K_M, the quality degradation starts to outweigh the benefits of additional parameters, and you should consider dropping to a smaller model at Q4_K_M instead. For critical applications, always test the quantized model on your specific use case. Some tasks are more sensitive to quantization than others. Math and coding tasks tend to degrade faster than creative writing or summarization. If you notice quality issues, try one step up in quantization before switching to a different model entirely.

# Calculate approximate model size
# Formula: parameters (B) * bits_per_weight / 8 = size in GB
# Example: 70B at Q4_K_M (4.5 bpw)
# 70 * 4.5 / 8 = ~39 GB

# Download a specific quantization from HuggingFace
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-GGUF \
  Llama-3.3-70B-Instruct-Q4_K_M.gguf --local-dir models/

# Run with llama.cpp specifying the quant directly
./llama-cli -m models/Llama-3.3-70B-Instruct-Q4_K_M.gguf -ngl 81 -c 4096

Frequently asked questions

Does quantization affect response speed?: Yes, and in a positive direction. Lower-precision weights mean fewer bytes to transfer from memory per token, so quantized models often generate tokens faster than their FP16 counterparts. A 70B model at Q4_K_M can generate 20-40% more tokens per second than at FP16 on the same hardware, assuming the FP16 version even fits in memory. The speed benefit diminishes at higher quantization levels.
Can I quantize a model myself?: Yes. The llama.cpp project includes a quantize tool that converts FP16 GGUF files to any supported quantization level. Download the FP16 GGUF from Hugging Face, then run the quantize binary with your target format. However, most popular models already have pre-quantized versions available, so manual quantization is only necessary for new or obscure models.
Is Q4_K_M always the best default choice?: For most users and most models, yes. It offers the best balance of quality preservation, memory efficiency, and ecosystem support. The main exceptions are: when you have ample VRAM and want to maximize quality (use Q5_K_M or Q6_K), when you need to squeeze a much larger model into limited VRAM (use Q3_K_M), or when you are on Blackwell hardware (try NVFP4).
Do different models respond differently to quantization?: Yes. Models with more parameters tend to be more resilient to quantization because they have more redundancy in their weights. A 70B model at Q4_K_M loses proportionally less quality than a 7B model at Q4_K_M. Models with mixture-of-experts architecture also tend to handle quantization well because only a subset of parameters is active for any given token. The training methodology matters too: models trained with quantization-aware techniques degrade less gracefully at lower bit widths.
What is the difference between Q4_K_S and Q4_K_M?: Both target approximately 4 bits per weight, but Q4_K_M (Medium) assigns higher precision to more sensitive layers than Q4_K_S (Small). Q4_K_M is about 5-10% larger but preserves slightly more quality. In practice, the difference is small enough that most users default to Q4_K_M for the marginal quality benefit. Q4_K_S is useful when you are right at the edge of fitting and need to save every possible megabyte.