DeepSeek V3 671B
DeepSeek V3 671B needs roughly 458.3 GB VRAM at Q4_K_M quantization (1503.6 GB at FP16). 3 GPUs we track can run it fully in VRAM at 8k context.
3 GPUs run this natively · 0 with CPU offload
DeepSeek V3 671B is a Mixture of Experts (MoE) model with 671B total parameters but only 37B active per token developed by DeepSeek. December 2024 671B MoE with Multi-Head Latent Attention (MLA) — 16× smaller KV cache than standard GQA.
To run DeepSeek V3 671B locally: Full model needs ~300GB+ at Q4. Q2_K (~150GB) is the only local option for extreme builds. MLA helps but doesn't solve the VRAM problem. As a MoE model, inference speed depends on active parameters (37B) rather than total size.
MMLU-Pro 75.9%, Math 90.2% — frontier-class quality with efficient attention.
VRAM at each quantization
DeepSeek V3 671B natively supports a longer context window, but the table below is capped at 8k for comparability — KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP32 | 2684.0 GB | 0.51 GB | 3006.7 GB |
| BF16 | 1342.0 GB | 0.51 GB | 1503.6 GB |
| FP16 | 1342.0 GB | 0.51 GB | 1503.6 GB |
| Q8_0 | 713.3 GB | 0.51 GB | 799.4 GB |
| Q6_K | 550.9 GB | 0.51 GB | 617.6 GB |
| Q5_K_M | 477.8 GB | 0.51 GB | 535.7 GB |
| Q4_K_M | 408.6 GB | 0.51 GB | 458.3 GB |
| Q3_K_M | 322.8 GB | 0.51 GB | 362.1 GB |
| Q2_Krec | 255.7 GB | 0.51 GB | 286.9 GB |
| NVFP4cuda | 335.5 GB | 0.51 GB | 376.3 GB |
KV cache is calculated at 8k context (FP16). Note that NVFP4 only runs on CUDA GPUs. Turn on TurboQuant in the calculator above for lower KV cache estimates.
Benchmarks
GPUs that run DeepSeek V3 671B natively (3)
- Apple M4 Ultra (384GB)Q3_K_M · 14.6 t/s
- Apple M3 Ultra (512GB)Q4_K_M · 8.7 t/s
- Apple M2 Ultra (384GB)Q3_K_M · 10.7 t/s
Notes
MoE with MLA attention — KV cache is ~16× smaller than standard GQA; kvHeads/headDim here approximates MLA storage.
Compare DeepSeek V3 671B with other models
How to run DeepSeek V3 671B locally
Q2_K needs 286.9 GB — needs multiple datacenter-class GPUs (80 GB+ each).
Ollama
ollama run deepseek-v3llama.cpp
./llama-cli -m deepseek-v3.Q2_K.gguf -c 8192 -ngl 99LM Studio: DeepSeek V3 is a 671B MoE model. Even at Q2, it requires significant VRAM. Check LM Studio for available GGUF splits if you have a multi-GPU setup.
Why this quantization? DeepSeek V3 has 671B total parameters in a Mixture-of-Experts architecture with only 37B active per token. The MoE design means all 671B parameters must be loaded into VRAM even though only a fraction is used per forward pass. Q2 is the recommended quantization because it is the only level that makes the model remotely feasible on high-end multi-GPU setups. The MLA (Multi-head Latent Attention) architecture keeps the KV cache exceptionally small, partially offsetting the massive weight footprint.
Who is DeepSeek V3 671B for?
Researchers and organizations with multi-GPU servers (4+ high-VRAM GPUs) who want frontier-level open-weight performance. DeepSeek V3 matches or exceeds many commercial APIs on benchmarks, making it attractive for teams building self-hosted inference endpoints to replace paid API dependencies.
Best for
- Frontier-level general-purpose reasoning and analysis (MMLU-Pro: 75.9, GPQA: 59.1)
- Advanced mathematics and quantitative problem solving (MATH: 90.2)
- Code generation with competitive LiveCodeBench scores (40.5)
- Self-hosted API replacement for commercial LLM services
- Research evaluations requiring open-weight frontier models
Not ideal for
- Any setup with less than 4x 80 GB GPUs -- even at Q2, the model needs roughly 250 GB for weights
- Interactive use cases where fast time-to-first-token matters -- MoE routing adds latency
- Budget-conscious deployments -- consider DeepSeek R1 Distill 32B for 95% of use cases
Continue reading
Frequently asked questions
- What are the VRAM requirements for DeepSeek V3 671B?
- DeepSeek V3 671B requires approximately 458.2 GB of VRAM at Q4_K_M quantization, 799.4 GB at Q8, and 1503.6 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
- How many parameters does DeepSeek V3 671B have?
- DeepSeek V3 671B has 671 billion total parameters, but only 37 billion are active per token thanks to its Mixture of Experts (MoE) architecture. This makes inference significantly faster than the total parameter count suggests.
- How capable is DeepSeek V3 671B?
- DeepSeek V3 671B achieves an MMLU-Pro score of 75.9, placing it among the most capable open-weight models available — competitive with frontier systems on general knowledge and reasoning.
- Can DeepSeek V3 671B run on a 16 GB GPU?
- No. At Q4_K_M, DeepSeek V3 671B needs 458.2 GB of VRAM — more than 16 GB. You will need a multi-GPU server.
- Can DeepSeek V3 671B run on a 24 GB GPU?
- No. Even at Q4_K_M, DeepSeek V3 671B needs 458.2 GB. Consider a multi-GPU server with 80 GB+ total VRAM.
- What is the smallest quantization for DeepSeek V3 671B that fits in 24 GB of VRAM?
- DeepSeek V3 671B cannot fit in 24 GB of VRAM at any standard quantization level. The minimum needed is 286.9 GB at Q2_K.
- What GPU do I need to run DeepSeek V3 671B locally?
- You need a multi-GPU server. At Q4_K_M, DeepSeek V3 671B needs 458.2 GB VRAM, more than any single consumer GPU. Consider 2–4× H100 or A100 GPUs.