Mixtral 8x7B Instruct v0.1
Mixtral 8x7B Instruct v0.1 needs roughly 23.4GB VRAM at Q4 quantization (93.4GB at FP16). 41 GPUs we track can run it fully in VRAM at 8k context.
Mistral AI46.7B params12.9B active (MoE)32k contextApache 2.0Commercial use ok
VRAM at each quantization
Assumes 8k context. KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP16 | 93.4 GB | 1.07 GB | 105.8 GB |
| Q8 | 46.7 GB | 1.07 GB | 53.5 GB |
| Q6_K | 35.0 GB | 1.07 GB | 40.4 GB |
| Q5_K_M | 29.2 GB | 1.07 GB | 33.9 GB |
| Q4_K_M | 23.4 GB | 1.07 GB | 27.4 GB |
| Q3_K_M | 18.7 GB | 1.07 GB | 22.1 GB |
| Q2_K | 14.0 GB | 1.07 GB | 16.9 GB |
Benchmarks
GPUs that run Mixtral 8x7B Instruct v0.1 natively (41)
- NVIDIA RTX 5090Q4_K_M · 305.6 t/s
- NVIDIA RTX 4090Q3_K_M · 214.9 t/s
- NVIDIA RTX 3090Q3_K_M · 199.5 t/s
- NVIDIA RTX 3090 TiQ3_K_M · 214.9 t/s
- NVIDIA H100 80GBQ8 · 285.7 t/s
- NVIDIA A100 80GBQ8 · 173.9 t/s
- NVIDIA A100 40GBQ5_K_M · 212.2 t/s
- NVIDIA L40SQ6_K · 98.2 t/s
- NVIDIA RTX A6000Q6_K · 87.3 t/s
- NVIDIA RTX 6000 AdaQ6_K · 109.1 t/s
- AMD Radeon RX 7900 XTXQ3_K_M · 204.7 t/s
- AMD Radeon RX 7900 XTQ2_K · 227.4 t/s
- AMD Instinct MI300XFP16 · 226 t/s
- Apple M4 Ultra (384GB)FP16 · 46.6 t/s
- Apple M4 Ultra (192GB)FP16 · 46.6 t/s
- Apple M4 Max (128GB)FP16 · 23.3 t/s
- Apple M4 Max (96GB)Q8 · 46.6 t/s
- Apple M4 Max (64GB)Q8 · 46.6 t/s
- Apple M4 Max (48GB)Q6_K · 62.1 t/s
- Apple M4 Pro (48GB)Q6_K · 31 t/s
- Apple M4 Pro (24GB)Q2_K · 77.6 t/s
- Apple M4 (32GB)Q4_K_M · 20.5 t/s
- Apple M3 Max (128GB)FP16 · 17.1 t/s
- Apple M3 Max (96GB)Q8 · 34.1 t/s
- Apple M3 Max (64GB)Q8 · 34.1 t/s
- Apple M3 Max (48GB)Q6_K · 45.5 t/s
- Apple M3 Max (36GB)Q4_K_M · 68.2 t/s
- Apple M3 Pro (36GB)Q4_K_M · 25.6 t/s
- Apple M3 (24GB)Q2_K · 28.4 t/s
- Apple M2 Ultra (384GB)FP16 · 34.1 t/s
- Apple M2 Ultra (192GB)FP16 · 34.1 t/s
- Apple M2 Max (96GB)Q8 · 34.1 t/s
- Apple M2 Max (64GB)Q8 · 34.1 t/s
- Apple M2 Max (32GB)Q4_K_M · 68.2 t/s
- Apple M2 Pro (32GB)Q4_K_M · 34.1 t/s
- Apple M2 (24GB)Q2_K · 28.4 t/s
- Apple M1 Ultra (128GB)FP16 · 34.1 t/s
- Apple M1 Ultra (64GB)Q8 · 68.2 t/s
- Apple M1 Max (64GB)Q8 · 34.1 t/s
- Apple M1 Max (32GB)Q4_K_M · 68.2 t/s
- Apple M1 Pro (32GB)Q4_K_M · 34.1 t/s
Plus 10 GPUs that run it with CPU offload (slower)
- NVIDIA RTX 4080Q6_K · 18.5 t/s
- NVIDIA RTX 4070 TiQ5_K_M · 15.6 t/s
- NVIDIA RTX 4070Q5_K_M · 15.6 t/s
- NVIDIA RTX 4060 Ti 16GBQ6_K · 7.4 t/s
- NVIDIA RTX 4060Q4_K_M · 10.5 t/s
- NVIDIA RTX 3080 10GBQ5_K_M · 23.6 t/s
- NVIDIA RTX 3060 12GBQ5_K_M · 11.2 t/s
- AMD Radeon RX 6800 XTQ6_K · 13.2 t/s
- Intel Arc A770 16GBQ6_K · 14.5 t/s
- CPU only (system RAM)Q4_K_M · 1.6 t/s
Notes
MoE: 47B total params, only ~13B active per token — fast if it fits.