Mixtral 8x7B Instruct v0.1
Mixtral 8x7B Instruct v0.1 needs roughly 30.6 GB VRAM at Q4_K_M quantization (105.8 GB at FP16). 67 GPUs we track can run it fully in VRAM at 8k context.
67 GPUs run this natively · 27 with CPU offload
Mistral AI46.7B params12.9B active (MoE)32k contextApache 2.0Commercial use ok
Mixtral 8x7B Instruct v0.1 is a Mixture of Experts (MoE) model with 46.7B total parameters but only 12.9B active per token developed by Mistral AI. December 2023 MoE pioneer — 46.7B total parameters with ~13B active per token via 8 experts and top-2 routing.
To run Mixtral 8x7B Instruct v0.1 locally: Memory costs proportional to 47B (not 13B active). Q4_K_M ~28-32GB — fits on 24GB GPU with tight context or 32GB+ recommended. Community papers show successful mixed quantization on desktop hardware. As a MoE model, inference speed depends on active parameters (12.9B) rather than total size.
Outperforms Llama-2-70B and GPT-3.5 on most benchmarks. MMLU 70.6%, GSM8K 74.4%. MT Bench 8.30 beats GPT-3.5 Turbo.
VRAM at each quantization
Assumes 8k context. KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP32 | 186.8 GB | 1.07 GB | 210.4 GB |
| BF16 | 93.4 GB | 1.07 GB | 105.8 GB |
| FP16 | 93.4 GB | 1.07 GB | 105.8 GB |
| Q8_0 | 46.7 GB | 1.07 GB | 53.5 GB |
| Q6_K | 38.3 GB | 1.07 GB | 44.1 GB |
| Q5_K_M | 30.1 GB | 1.07 GB | 34.9 GB |
| Q4_K_Mrec | 26.3 GB | 1.07 GB | 30.6 GB |
| Q3_K_M | 20.1 GB | 1.07 GB | 23.7 GB |
| Q2_K | 15.4 GB | 1.07 GB | 18.4 GB |
| NVFP4cuda | 23.4 GB | 1.07 GB | 27.4 GB |
KV cache shown at 8k context (FP16). NVFP4 requires a CUDA GPU. Enable TurboQuant in the calculator to see reduced KV cache estimates.
Benchmarks
GPUs that run Mixtral 8x7B Instruct v0.1 natively (67)
- NVIDIA RTX 5090NVFP4 · 305.6 t/s
- NVIDIA RTX 4090Q2_K · 261.3 t/s
- NVIDIA RTX 3090Q2_K · 242.6 t/s
- NVIDIA RTX 3090 TiQ2_K · 261.3 t/s
- NVIDIA H100 80GBNVFP4 · 571.3 t/s
- NVIDIA A100 80GBNVFP4 · 347.7 t/s
- NVIDIA A100 40GBNVFP4 · 265.2 t/s
- NVIDIA L40SNVFP4 · 147.3 t/s
- NVIDIA RTX A6000NVFP4 · 131 t/s
- NVIDIA RTX 4000 AdaQ2_K · 82.9 t/s
- NVIDIA RTX 4500 AdaQ2_K · 112 t/s
- NVIDIA RTX 5000 AdaNVFP4 · 98.2 t/s
- NVIDIA RTX 6000 AdaNVFP4 · 163.7 t/s
- NVIDIA RTX Pro 6000NVFP4 · 229.2 t/s
- NVIDIA DGX Spark (128GB)BF16 · 11.6 t/s
- AMD Radeon RX 7900 XTXQ2_K · 248.8 t/s
- AMD Radeon RX 7900 XTQ2_K · 207.3 t/s
- AMD Radeon PRO W7800Q3_K_M · 114.2 t/s
- AMD Radeon PRO W7900Q6_K · 89.8 t/s
- AMD Instinct MI300XBF16 · 226 t/s
- AMD Radeon AI Pro 9700 32GBQ3_K_M · 126.9 t/s
- AMD Strix Halo (128GB)BF16 · 10.9 t/s
- AMD Strix Halo (96GB)Q8_0 · 21.8 t/s
- AMD Strix Halo (64GB)Q8_0 · 21.8 t/s
- Apple M5 Max (128GB)BF16 · 26.2 t/s
- Apple M5 Max (64GB)Q8_0 · 52.4 t/s
- Apple M5 Max (48GB)Q5_K_M · 81.3 t/s
- Apple M5 Pro (48GB)Q5_K_M · 40.6 t/s
- Apple M5 Pro (36GB)Q4_K_M · 46.5 t/s
- Apple M5 Pro (24GB)Q2_K · 79.6 t/s
- Apple M5 (32GB)Q3_K_M · 30.3 t/s
- Apple M4 Ultra (384GB)FP32 · 23.3 t/s
- Apple M4 Ultra (192GB)BF16 · 46.6 t/s
- Apple M4 Max (128GB)BF16 · 23.3 t/s
- Apple M4 Max (96GB)Q8_0 · 46.6 t/s
- Apple M4 Max (64GB)Q8_0 · 46.6 t/s
- Apple M4 Max (48GB)Q5_K_M · 72.3 t/s
- Apple M4 Pro (48GB)Q5_K_M · 36.1 t/s
- Apple M4 Pro (24GB)Q2_K · 70.8 t/s
- Apple M4 (32GB)Q3_K_M · 23.8 t/s
- Apple M3 Ultra (512GB)FP32 · 17.5 t/s
- Apple M3 Ultra (256GB)FP32 · 17.5 t/s
- Apple M3 Ultra (96GB)Q8_0 · 69.8 t/s
- Apple M3 Max (128GB)BF16 · 17.1 t/s
- Apple M3 Max (96GB)Q8_0 · 34.1 t/s
- Apple M3 Max (64GB)Q8_0 · 34.1 t/s
- Apple M3 Max (48GB)Q5_K_M · 53 t/s
- Apple M3 Max (36GB)Q4_K_M · 60.6 t/s
- Apple M3 Pro (36GB)Q4_K_M · 22.7 t/s
- Apple M3 (24GB)Q2_K · 25.9 t/s
- Apple M2 Ultra (384GB)FP32 · 17.1 t/s
- Apple M2 Ultra (192GB)BF16 · 34.1 t/s
- Apple M2 Max (96GB)Q8_0 · 34.1 t/s
- Apple M2 Max (64GB)Q8_0 · 34.1 t/s
- Apple M2 Max (32GB)Q3_K_M · 79.3 t/s
- Apple M2 Pro (32GB)Q3_K_M · 39.7 t/s
- Apple M2 (24GB)Q2_K · 25.9 t/s
- Apple M1 Ultra (128GB)BF16 · 34.1 t/s
- Apple M1 Ultra (64GB)Q8_0 · 68.2 t/s
- Apple M1 Max (64GB)Q8_0 · 34.1 t/s
- Apple M1 Max (32GB)Q3_K_M · 79.3 t/s
- Apple M1 Pro (32GB)Q3_K_M · 39.7 t/s
- Intel Arc Pro B70 24GBQ2_K · 118.2 t/s
- Intel Arc Pro B60 24GBQ2_K · 98.5 t/s
- Intel Data Center GPU Max 1550BF16 · 139.7 t/s
- Intel Data Center GPU Max 1100Q6_K · 127.8 t/s
- Intel Arc 140V (32GB)Q3_K_M · 27.2 t/s
Plus 27 GPUs that run it with CPU offload (slower)
- NVIDIA RTX 5080NVFP4 · 37.2 t/s
- NVIDIA RTX 5070 TiNVFP4 · 34.7 t/s
- NVIDIA RTX 5070NVFP4 · 26 t/s
- NVIDIA RTX 5060 Ti 16GBNVFP4 · 17.4 t/s
- NVIDIA RTX 5060NVFP4 · 17.4 t/s
- NVIDIA RTX 5050NVFP4 · 12.4 t/s
- NVIDIA RTX 4080NVFP4 · 27.8 t/s
- NVIDIA RTX 4070 TiNVFP4 · 19.5 t/s
- NVIDIA RTX 4070NVFP4 · 19.5 t/s
- NVIDIA RTX 4060 Ti 16GBNVFP4 · 11.2 t/s
- NVIDIA RTX 4060NVFP4 · 10.5 t/s
- NVIDIA RTX 3080 10GBNVFP4 · 29.5 t/s
- NVIDIA RTX 3060 12GBNVFP4 · 14 t/s
- AMD Radeon RX 7900 GREQ5_K_M · 17.3 t/s
- AMD Radeon RX 6800 XTQ5_K_M · 15.4 t/s
- Intel Arc B580 12GBQ5_K_M · 13.7 t/s
- Intel Arc B570 10GBQ5_K_M · 11.4 t/s
- Intel Arc A770 16GBQ5_K_M · 16.9 t/s
- Intel Arc A770 8GBQ4_K_M · 17.6 t/s
- Intel Arc A750 8GBQ4_K_M · 17.6 t/s
- Intel Arc A580 8GBQ4_K_M · 17.6 t/s
- Intel Arc A380 6GBQ4_K_M · 6.4 t/s
- Intel Arc A310 4GBQ3_K_M · 5.6 t/s
- Intel Arc Pro A60 12GBQ5_K_M · 11.6 t/s
- Intel Arc Pro A50 6GBQ4_K_M · 6.6 t/s
- Intel Arc Pro A40 6GBQ4_K_M · 6.6 t/s
- CPU only (system RAM)Q3_K_M · 1.9 t/s
Notes
MoE: 47B total params, only ~13B active per token — fast if it fits.
Compare Mixtral 8x7B Instruct v0.1 with other models
Frequently asked questions
- What are the VRAM requirements for Mixtral 8x7B Instruct v0.1?
- Mixtral 8x7B Instruct v0.1 requires approximately 30.6 GB of VRAM at Q4_K_M quantization, 53.5 GB at Q8, and 105.8 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
- How many parameters does Mixtral 8x7B Instruct v0.1 have?
- Mixtral 8x7B Instruct v0.1 has 46.7 billion total parameters, but only 12.9 billion are active per token thanks to its Mixture of Experts (MoE) architecture. This makes inference significantly faster than the total parameter count suggests.
- How capable is Mixtral 8x7B Instruct v0.1?
- Mixtral 8x7B Instruct v0.1 has an MMLU-Pro score of 29.7, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
- Can Mixtral 8x7B Instruct v0.1 run on a 16 GB GPU?
- No. At Q4_K_M, Mixtral 8x7B Instruct v0.1 needs 30.6 GB of VRAM — more than 16 GB. You will need a 48 GB GPU like the RTX 6000 Ada or a dual-GPU setup.
- Can Mixtral 8x7B Instruct v0.1 run on a 24 GB GPU?
- No. Even at Q4_K_M, Mixtral 8x7B Instruct v0.1 needs 30.6 GB. Consider a 48 GB card like the RTX 6000 Ada or a dual RTX 4090 setup.
- What is the smallest quantization for Mixtral 8x7B Instruct v0.1 that fits in 24 GB of VRAM?
- At Q3_K_M, Mixtral 8x7B Instruct v0.1 needs 23.7 GB — the highest-quality quantization that fits in 24 GB of VRAM.
- What GPU do I need to run Mixtral 8x7B Instruct v0.1 locally?
- You need a 48 GB GPU or a dual-GPU setup. At Q4_K_M, Mixtral 8x7B Instruct v0.1 needs 30.6 GB VRAM. Options: RTX 6000 Ada (48 GB), A6000 (48 GB), or 2× RTX 4090.