CanItRun Logocanitrun.

Mistral Small 22B

Mistral Small 22B needs roughly 17.3 GB VRAM at Q4_K_M quantization (51.8 GB at FP16). 84 GPUs we track can run it fully in VRAM at 8k context.

84 GPUs run this natively · 21 with CPU offload

Mistral AI22.2B params32k contextMistral ResearchNon-commercial only

Mistral Small 22B is a 22.2B parameter dense model developed by Mistral AI. September 2024 22B model with strong general capabilities.

To run Mistral Small 22B locally: Q4_K_M needs ~17.3GB total once KV cache and overhead are counted, past a 16GB card's ~15.2GB effective capacity -- a 24GB GPU is the real minimum, with comfortable headroom to spare.

MMLU-Pro 49.2%, HumanEval 81.1%, solid mid-range performer.

VRAM at each quantization

Numbers here are computed at 8k context. Because KV cache grows linearly with context length, expect higher totals at longer sequence lengths.

QuantWeightsKV cacheTotal
FP3288.8 GB1.88 GB101.6 GB
BF1644.4 GB1.88 GB51.8 GB
FP1644.4 GB1.88 GB51.8 GB
Q8_023.6 GB1.88 GB28.5 GB
Q6_K18.2 GB1.88 GB22.5 GB
Q5_K_M15.8 GB1.88 GB19.8 GB
Q4_K_Mrec13.5 GB1.88 GB17.3 GB
Q3_K_M10.7 GB1.88 GB14.1 GB
Q2_K8.5 GB1.88 GB11.6 GB
NVFP4cuda11.1 GB1.88 GB14.5 GB

KV cache figures assume 8k context at FP16. NVFP4 quantization requires a CUDA-capable GPU. Enable TurboQuant in the calculator to see reduced KV cache estimates.

Benchmarks

MMLU-Pro
49.2
GPQA Diamond
44.0
IFEval
72.8
MATH
70.0

GPUs that run Mistral Small 22B natively (84)

Show 79 more
Plus 21 GPUs that run it with CPU offload (slower)
Hugging Face ↗Ollama ↗Released 2024-09-18

Continue reading

tutorials10 min
GGUF Quantization Explained: Q4, Q5, Q6, Q8 Compared
comparisons9 min
GGUF vs EXL2 vs AWQ: Which Quantization Format to Use
vram-guides9 min
Mistral & Mixtral Family: Complete VRAM Reference

Frequently asked questions

What are the VRAM requirements for Mistral Small 22B?
Mistral Small 22B requires approximately 17.3 GB of VRAM at Q4_K_M quantization, 28.5 GB at Q8, and 51.8 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
How many parameters does Mistral Small 22B have?
Mistral Small 22B has 22.2 billion parameters.
How capable is Mistral Small 22B?
Mistral Small 22B has an MMLU-Pro score of 49.2, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
Can Mistral Small 22B run on a 16 GB GPU?
No. At Q4_K_M, Mistral Small 22B needs 17.3 GB of VRAM, more than 16 GB. You will need a 24 GB GPU like the RTX 4090 or RTX 3090.
Can Mistral Small 22B run on a 24 GB GPU?
Yes. Mistral Small 22B fits in a 24 GB GPU at Q4_K_M, requiring 17.3 GB VRAM. GPUs with 24 GB include the RTX 4090, RTX 3090, and RTX 3090 Ti.
What is the smallest quantization for Mistral Small 22B that fits in 24 GB of VRAM?
At NVFP4, Mistral Small 22B needs 14.5 GB, the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Mistral Small 22B locally?
A 24 GB GPU is the minimum. At Q4_K_M, Mistral Small 22B needs 17.3 GB VRAM. Good options: RTX 4090 (24 GB), RTX 3090 (24 GB).