GGUF vs EXL2 vs AWQ: Which Quantization Format to Use

Three Formats, Three Trade-offs

The local LLM ecosystem has converged on three main quantization formats, each with distinct strengths. GGUF is the universal format: it works on every platform (NVIDIA, AMD, Apple Silicon, Intel, CPU-only) through llama.cpp and its derivatives. EXL2 is the quality-optimized format for NVIDIA GPUs: it extracts the best possible output quality at any given bits-per-weight through ExLlamaV2's sophisticated per-layer calibration. AWQ (Activation-Aware Weight Quantization) is the speed-focused format for NVIDIA GPUs: it is optimized for fast inference throughput, particularly in serving scenarios. All three formats achieve the same fundamental goal of compressing model weights to fit in less memory, but they take different technical approaches and optimize for different metrics. Understanding these differences helps you choose the right format for your specific hardware and use case rather than defaulting to whatever the first search result suggests.

GGUF: The Universal Format

GGUF is the file format used by llama.cpp and is by far the most widely supported quantization format. Its key advantage is universality: a GGUF file runs on NVIDIA GPUs via CUDA, AMD GPUs via ROCm or Vulkan, Apple Silicon via Metal, Intel GPUs via SYCL, and even on CPU-only systems. This cross-platform support makes GGUF the safe default choice. If you are unsure which format to use, GGUF is the answer. The K-quant system (Q2_K through Q8_0) uses mixed-precision quantization that assigns different bit-widths to different model layers based on their importance. This produces better quality than naive uniform quantization at the same average bits-per-weight. GGUF files are also self-contained: they include weights, tokenizer, and model metadata in a single file, making them easy to distribute and manage. The llama.cpp project is actively developed with contributions from hundreds of developers. New model architectures are typically supported in GGUF within days of release. Quantization improvements like importance-matrix calibration (imatrix) have narrowed the quality gap with more specialized formats. For most users on most hardware, GGUF through Ollama or llama.cpp is the right choice.

# Download a GGUF model from HuggingFace
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-GGUF \
  Llama-3.3-70B-Instruct-Q4_K_M.gguf --local-dir models/

# Run with llama.cpp
./llama-cli -m models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  -ngl 81 -c 4096

# Or import into Ollama
ollama create my-llama -f Modelfile  # FROM ./model.gguf

EXL2: Maximum Quality on NVIDIA

EXL2 is the quantization format used by ExLlamaV2, a CUDA-only inference engine focused on maximum quality at any given bit-width. EXL2's standout feature is per-layer calibration: during quantization, the tool runs calibration data through the model and measures how much each layer contributes to output quality. Layers that matter more receive higher precision, and less critical layers are compressed more aggressively. The result is measurably better output quality than GGUF at the same average bits-per-weight, particularly at low bit-widths (2.5-4.0 bpw) where the quality differences are most significant. EXL2 also supports arbitrary bit-widths, not just the fixed Q2/Q3/Q4/Q5/Q6/Q8 steps of GGUF. You can request 3.5 bpw, 4.25 bpw, or any other target, and EXL2 will allocate precision across layers to hit that target while maximizing quality. This granular control is valuable when you have a precise VRAM budget and want to use every megabyte optimally. The trade-offs are significant: EXL2 is NVIDIA-only (requires CUDA), has a smaller ecosystem than GGUF, and ExLlamaV2 supports fewer model architectures. It is also harder to set up, requiring Python, CUDA toolkit, and manual configuration rather than Ollama's one-command experience. Use EXL2 when you have an NVIDIA GPU, you care about squeezing maximum quality from limited VRAM, and you are comfortable with a more hands-on setup process.

# Install ExLlamaV2
pip install exllamav2

# Run an EXL2 model
python -m exllamav2.chat \
  --model_dir models/Llama-3.3-70B-Instruct-exl2-4.0bpw/ \
  --mode chat

# Or use TabbyAPI for an OpenAI-compatible server
# (TabbyAPI wraps ExLlamaV2 with an API layer)
python -m tabbyapi.main --model-name Llama-3.3-70B-Instruct-exl2-4.0bpw

AWQ: Speed-Optimized Inference

AWQ (Activation-Aware Weight Quantization) takes a different approach to quantization. Instead of calibrating precision by layer importance (like EXL2) or using mixed-precision groups (like GGUF K-quants), AWQ identifies which weights are most important based on activation magnitudes and preserves those at higher precision. The key insight is that a small percentage of weights (roughly 1%) disproportionately affect model output, and protecting those weights during quantization preserves most of the model quality. AWQ models are typically quantized to 4 bits and are designed for fast inference throughput. They work with inference engines like vLLM, TGI (Text Generation Inference), and AutoAWQ, all of which are optimized for NVIDIA CUDA hardware. AWQ is particularly popular in serving scenarios where you need to handle multiple concurrent requests with high throughput, because its quantization scheme maps efficiently to GPU tensor core operations. For individual users running single-query inference, AWQ's speed advantage over GGUF is modest (10-20% faster prompt processing, similar generation speed). The format's strengths show more in multi-user serving where batched inference amortizes compute across requests. If you are setting up a personal inference server, GGUF or EXL2 are better choices. If you are serving models to multiple users via an API, AWQ with vLLM is worth evaluating.

# Install AutoAWQ for running AWQ models
pip install autoawq

# Download an AWQ model
huggingface-cli download TheBloke/Llama-3-70B-AWQ \
  --local-dir models/Llama-3-70B-AWQ/

# Serve with vLLM for high-throughput inference
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model models/Llama-3-70B-AWQ/ \
  --quantization awq \
  --dtype half

Quality Comparison at the Same Bit-Width

At 4.0 bits per weight, the quality ranking from controlled benchmarks is generally: EXL2 (best) > GGUF Q4_K_M (close second) > AWQ (comparable to GGUF). The gap between EXL2 and GGUF has narrowed over time as llama.cpp has added importance-matrix quantization, but EXL2's per-layer calibration still provides a measurable edge, particularly on models below 7B parameters where every bit of precision matters more. At higher bit-widths (5.0+ bpw), the quality differences become negligible. EXL2 at 5.0 bpw, GGUF Q5_K_M, and AWQ at 4-bit with group size 32 all produce output very close to FP16 baseline. The practical implication is that format choice matters most when you are in the 2.5-4.0 bpw range, which is exactly where VRAM-constrained users operate. If you have enough VRAM for Q5_K_M or higher, the format differences are academic and you should choose based on ecosystem compatibility. One nuance: quality comparisons depend on the specific model architecture. Some models quantize better in one format than another due to their weight distributions. If you are making a critical deployment decision, benchmark your specific model and use case in both formats rather than relying on general comparisons.

Hardware Compatibility Matrix

GGUF has universal hardware support. It runs on NVIDIA GPUs (CUDA), AMD GPUs (ROCm on Linux, Vulkan on all platforms), Apple Silicon (Metal), Intel GPUs (SYCL), and CPU-only systems. This makes it the only option for Mac users, AMD GPU users on Windows, Intel GPU users, and anyone running without a GPU. EXL2 is strictly NVIDIA CUDA only. It requires a CUDA-capable GPU with compute capability 6.0 or higher (Pascal generation and newer). It does not work on AMD, Apple Silicon, Intel, or CPU-only setups. If you are not on NVIDIA hardware, EXL2 is not an option. AWQ has the same NVIDIA restriction as EXL2 in practice. While the quantization method is theoretically hardware-agnostic, the inference engines that support AWQ (vLLM, AutoAWQ) are all CUDA-focused. Some experimental support exists for other backends, but it is not production-ready. For users with NVIDIA GPUs, all three formats are available. For everyone else, GGUF is the answer. This hardware lock-in is the single biggest reason GGUF dominates market share despite EXL2's quality advantage: the majority of local LLM users include Mac users and AMD GPU owners who have no alternative.

Ecosystem and Community Support

The ecosystem around each format differs substantially. GGUF benefits from the llama.cpp project's momentum: it is the most actively developed open-source inference engine, with hundreds of contributors and near-instant support for new model architectures. Ollama, the most popular LLM runner, uses GGUF exclusively. Model quantizers like TheBloke and bartowski on Hugging Face provide GGUF files for virtually every new model release within hours. The community is large and responsive, meaning troubleshooting help is readily available. EXL2 has a smaller but dedicated community centered around ExLlamaV2 and TabbyAPI. The quantization process is more involved (requiring calibration data and GPU time), so pre-quantized EXL2 models are less universally available. When they are available, they are often provided at multiple bit-widths (2.5, 3.0, 3.5, 4.0, 5.0, 6.0 bpw), giving you more granular control than GGUF's fixed quantization levels. AWQ's ecosystem is most mature in the production serving space. vLLM is the standard for high-throughput LLM serving, and AWQ integration is well-tested. For individual users, AWQ's ecosystem is less developed than GGUF. Documentation and community support exist but are more focused on deployment engineers than hobbyists.

Which Format Should You Use

For most users, the decision tree is simple. If you are on Mac, AMD GPU, Intel GPU, or CPU-only: use GGUF. There is no alternative, and GGUF is excellent. If you are on an NVIDIA GPU and want the easiest setup: use GGUF through Ollama. The one-command experience is hard to beat, and quality is within a few percent of the best alternatives. If you are on an NVIDIA GPU and want maximum quality at low bit-widths (2.5-4.0 bpw): use EXL2 through ExLlamaV2 or TabbyAPI. The quality advantage is real and worth the additional setup complexity if you are VRAM-constrained and quality-sensitive. If you are serving models to multiple users via an API: evaluate AWQ with vLLM. The batched inference performance is genuinely better than alternatives in serving scenarios. If you are unsure or just getting started: use GGUF with Ollama. You can always experiment with EXL2 later once you understand your needs. The format decision is not permanent: you can download the same model in different formats and compare output quality and speed on your specific hardware. Many model uploaders on Hugging Face provide the same model in GGUF, EXL2, and AWQ formats.

Format Selection Summary:

                 GGUF          EXL2          AWQ
Hardware     All platforms   NVIDIA only   NVIDIA only
Quality @4bpw  Very good      Best         Good
Speed          Good           Good          Best (batched)
Ease of use    Easiest        Moderate      Moderate
Ecosystem      Largest        Smaller       Serving-focused
Best for       Everyone       Quality       Multi-user serving
                              enthusiasts