Qwen 2.5 Coder 32B Instruct
Qwen 2.5 Coder 32B Instruct needs roughly 24.6 GB VRAM at Q4_K_M quantization (75.2 GB at FP16). 63 GPUs we track can run it fully in VRAM at 8k context.
63 GPUs run this natively · 27 with CPU offload
Qwen 2.5 Coder 32B Instruct is a 32.5B parameter dense model developed by Alibaba. November 2024 coding-specialized variant — best open-weight coding model at this size.
To run Qwen 2.5 Coder 32B Instruct locally: Same VRAM requirements as Qwen2.5-32B (~18-20GB Q4). The top choice for developers with 24GB GPUs.
HumanEval 92.7% is exceptional, rivaling much larger models. MMLU-Pro 50.4% shows strong general capabilities too.
VRAM at each quantization
Figures below assume 8k context; KV cache grows linearly as context length increases.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP32 | 130.0 GB | 2.15 GB | 148.0 GB |
| BF16 | 65.0 GB | 2.15 GB | 75.2 GB |
| FP16 | 65.0 GB | 2.15 GB | 75.2 GB |
| Q8_0 | 34.5 GB | 2.15 GB | 41.1 GB |
| Q6_K | 26.7 GB | 2.15 GB | 32.3 GB |
| Q5_K_M | 23.1 GB | 2.15 GB | 28.3 GB |
| Q4_K_Mrec | 19.8 GB | 2.15 GB | 24.6 GB |
| Q3_K_M | 15.6 GB | 2.15 GB | 19.9 GB |
| Q2_K | 12.4 GB | 2.15 GB | 16.3 GB |
| NVFP4cuda | 16.3 GB | 2.15 GB | 20.6 GB |
Shown at 8k context with FP16 KV cache. NVFP4 needs a CUDA GPU to run. Toggle TurboQuant in the calculator to view compressed KV cache numbers.
Benchmarks
GPUs that run Qwen 2.5 Coder 32B Instruct natively (63)
- NVIDIA RTX 5090NVFP4 · 63.3 t/s
- NVIDIA RTX 4090NVFP4 · 35.6 t/s
- NVIDIA RTX 3090NVFP4 · 33.1 t/s
- NVIDIA RTX 3090 TiNVFP4 · 35.6 t/s
- NVIDIA H100 80GBBF16 · 32.4 t/s
- NVIDIA A100 80GBBF16 · 19.7 t/s
- NVIDIA A100 40GBNVFP4 · 54.9 t/s
- NVIDIA L40SNVFP4 · 30.5 t/s
- NVIDIA RTX A6000NVFP4 · 27.1 t/s
- NVIDIA RTX 4000 AdaQ2_K · 14.3 t/s
- NVIDIA RTX 4500 AdaNVFP4 · 15.3 t/s
- NVIDIA RTX 5000 AdaNVFP4 · 20.4 t/s
- NVIDIA RTX 6000 AdaNVFP4 · 33.9 t/s
- NVIDIA RTX Pro 6000BF16 · 13 t/s
- NVIDIA DGX Spark (128GB)BF16 · 2.6 t/s
- AMD Radeon RX 7900 XTXQ3_K_M · 35.1 t/s
- AMD Radeon RX 7900 XTQ2_K · 35.8 t/s
- AMD Radeon PRO W7800Q5_K_M · 14.8 t/s
- AMD Radeon PRO W7900Q8_0 · 15.3 t/s
- AMD Instinct MI300XFP32 · 26.1 t/s
- AMD Radeon AI Pro 9700 32GBQ5_K_M · 16.5 t/s
- AMD Strix Halo (128GB)BF16 · 2.5 t/s
- AMD Strix Halo (96GB)BF16 · 2.5 t/s
- AMD Strix Halo (64GB)Q8_0 · 4.5 t/s
- Apple M5 Max (128GB)BF16 · 7.3 t/s
- Apple M5 Max (64GB)Q8_0 · 13.4 t/s
- Apple M5 Max (48GB)Q6_K · 17 t/s
- Apple M5 Pro (48GB)Q6_K · 8.5 t/s
- Apple M5 Pro (36GB)Q4_K_M · 11.2 t/s
- Apple M5 (32GB)Q3_K_M · 6.9 t/s
- Apple M4 Ultra (384GB)FP32 · 6.6 t/s
- Apple M4 Ultra (192GB)FP32 · 6.6 t/s
- Apple M4 Max (128GB)BF16 · 6.5 t/s
- Apple M4 Max (96GB)BF16 · 6.5 t/s
- Apple M4 Max (64GB)Q8_0 · 11.9 t/s
- Apple M4 Max (48GB)Q6_K · 15.2 t/s
- Apple M4 Pro (48GB)Q6_K · 7.6 t/s
- Apple M4 (32GB)Q3_K_M · 5.4 t/s
- Apple M3 Ultra (512GB)FP32 · 5 t/s
- Apple M3 Ultra (256GB)FP32 · 5 t/s
- Apple M3 Ultra (96GB)BF16 · 9.8 t/s
- Apple M3 Max (128GB)BF16 · 4.8 t/s
- Apple M3 Max (96GB)BF16 · 4.8 t/s
- Apple M3 Max (64GB)Q8_0 · 8.7 t/s
- Apple M3 Max (48GB)Q6_K · 11.1 t/s
- Apple M3 Max (36GB)Q4_K_M · 14.6 t/s
- Apple M3 Pro (36GB)Q4_K_M · 5.5 t/s
- Apple M2 Ultra (384GB)FP32 · 4.8 t/s
- Apple M2 Ultra (192GB)FP32 · 4.8 t/s
- Apple M2 Max (96GB)BF16 · 4.8 t/s
- Apple M2 Max (64GB)Q8_0 · 8.7 t/s
- Apple M2 Max (32GB)Q3_K_M · 18 t/s
- Apple M2 Pro (32GB)Q3_K_M · 9 t/s
- Apple M1 Ultra (128GB)BF16 · 9.5 t/s
- Apple M1 Ultra (64GB)Q8_0 · 17.4 t/s
- Apple M1 Max (64GB)Q8_0 · 8.7 t/s
- Apple M1 Max (32GB)Q3_K_M · 18 t/s
- Apple M1 Pro (32GB)Q3_K_M · 9 t/s
- Intel Arc Pro B70 24GBQ3_K_M · 16.7 t/s
- Intel Arc Pro B60 24GBQ3_K_M · 13.9 t/s
- Intel Data Center GPU Max 1550BF16 · 31.7 t/s
- Intel Data Center GPU Max 1100Q8_0 · 21.8 t/s
- Intel Arc 140V (32GB)Q3_K_M · 5 t/s
Plus 27 GPUs that run it with CPU offload (slower)
- NVIDIA RTX 5080NVFP4 · 6.6 t/s
- NVIDIA RTX 5070 TiNVFP4 · 6.5 t/s
- NVIDIA RTX 5070NVFP4 · 3.3 t/s
- NVIDIA RTX 5060 Ti 16GBNVFP4 · 5.6 t/s
- NVIDIA RTX 5060NVFP4 · 2.2 t/s
- NVIDIA RTX 5050NVFP4 · 2.1 t/s
- NVIDIA RTX 4080NVFP4 · 6.2 t/s
- NVIDIA RTX 4070 TiNVFP4 · 3.2 t/s
- NVIDIA RTX 4070NVFP4 · 3.2 t/s
- NVIDIA RTX 4060 Ti 16GBNVFP4 · 4.8 t/s
- NVIDIA RTX 4060NVFP4 · 2.1 t/s
- NVIDIA RTX 3080 10GBNVFP4 · 2.7 t/s
- NVIDIA RTX 3060 12GBNVFP4 · 3 t/s
- AMD Radeon RX 7900 GREQ8_0 · 1.1 t/s
- AMD Radeon RX 6800 XTQ8_0 · 1.1 t/s
- Intel Arc B580 12GBQ6_K · 1.4 t/s
- Intel Arc B570 10GBQ6_K · 1.2 t/s
- Intel Arc A770 16GBQ8_0 · 1.1 t/s
- Intel Arc A770 8GBQ6_K · 1.1 t/s
- Intel Arc A750 8GBQ6_K · 1.1 t/s
- Intel Arc A580 8GBQ6_K · 1.1 t/s
- Intel Arc A380 6GBQ5_K_M · 1.2 t/s
- Intel Arc A310 4GBQ5_K_M · 1.1 t/s
- Intel Arc Pro A60 12GBQ6_K · 1.3 t/s
- Intel Arc Pro A50 6GBQ5_K_M · 1.2 t/s
- Intel Arc Pro A40 6GBQ5_K_M · 1.2 t/s
- CPU only (system RAM)Q4_K_M · 1.8 t/s
Notes
Best open-weight coding model at this size.
Compare Qwen 2.5 Coder 32B Instruct with other models
How to run Qwen 2.5 Coder 32B Instruct locally
Q4_K_M needs 24.6 GB — needs a workstation or datacenter GPU (48–80 GB).
Ollama
ollama run qwen2.5-coder:32bllama.cpp
./llama-cli -m qwen2.5-coder-32b-instruct.Q4_K_M.gguf -c 8192 -ngl 99LM Studio: Search for 'Qwen 2.5 Coder 32B' in LM Studio. The Q4_K_M variant fits on a single 24 GB GPU, making it the top local coding model at this price point.
Why this quantization? At 32.5B dense parameters, Q4_K_M is the sweet spot for fitting on a single 24 GB consumer GPU (RTX 4090, RTX 3090). This model was specifically trained for code generation, so its performance is disproportionately strong for its size. Q4 keeps it within the 24 GB envelope while retaining enough precision for accurate code synthesis.
Who is Qwen 2.5 Coder 32B Instruct for?
Software developers who want the best local coding assistant that fits on a single high-end consumer GPU. Ideal for programmers who want to pair it with VS Code or a terminal-based workflow, and who value the Apache 2.0 license for integrating AI-assisted coding into commercial projects.
Best for
- Code generation, completion, and refactoring across popular languages
- Explaining and documenting existing codebases
- Unit test generation and bug detection
- Technical writing and API documentation
- Pair programming with local privacy -- your code never leaves your machine
Not ideal for
- General-purpose chat or creative writing -- this model is specialized for code
- Users with less than 20 GB of VRAM -- consider Qwen 2.5 7B Coder or DeepSeek R1 Distill 8B
- Tasks requiring strong multilingual natural-language understanding outside of code
Continue reading
Frequently asked questions
- What are the VRAM requirements for Qwen 2.5 Coder 32B Instruct?
- Qwen 2.5 Coder 32B Instruct requires approximately 24.6 GB of VRAM at Q4_K_M quantization, 41.1 GB at Q8, and 75.2 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
- How many parameters does Qwen 2.5 Coder 32B Instruct have?
- Qwen 2.5 Coder 32B Instruct has 32.5 billion parameters.
- How capable is Qwen 2.5 Coder 32B Instruct?
- With an MMLU-Pro score of 50.4, Qwen 2.5 Coder 32B Instruct delivers solid general-purpose performance suitable for most everyday tasks and professional use.
- Can Qwen 2.5 Coder 32B Instruct run on a 16 GB GPU?
- No. At Q4_K_M, Qwen 2.5 Coder 32B Instruct needs 24.6 GB of VRAM — more than 16 GB. You will need a 48 GB GPU like the RTX 6000 Ada or a dual-GPU setup.
- Can Qwen 2.5 Coder 32B Instruct run on a 24 GB GPU?
- No. Even at Q4_K_M, Qwen 2.5 Coder 32B Instruct needs 24.6 GB. Consider a 48 GB card like the RTX 6000 Ada or a dual RTX 4090 setup.
- What is the smallest quantization for Qwen 2.5 Coder 32B Instruct that fits in 24 GB of VRAM?
- At NVFP4, Qwen 2.5 Coder 32B Instruct needs 20.6 GB — the highest-quality quantization that fits in 24 GB of VRAM.
- What GPU do I need to run Qwen 2.5 Coder 32B Instruct locally?
- You need a 48 GB GPU or a dual-GPU setup. At Q4_K_M, Qwen 2.5 Coder 32B Instruct needs 24.6 GB VRAM. Options: RTX 6000 Ada (48 GB), A6000 (48 GB), or 2× RTX 4090.