CanItRun Logocanitrun.

Mistral Nemo 12B Instruct

Mistral Nemo 12B Instruct needs roughly 9.2 GB VRAM at Q4_K_M quantization (28.8 GB at FP16). 102 GPUs we track can run it fully in VRAM at 8k context.

102 GPUs run this natively · 5 with CPU offload

Mistral AI12.2B params125k contextApache 2.0Commercial use ok

Mistral Nemo 12B Instruct is a 12.2B parameter dense model developed by Mistral AI. July 2024 12B model with 128K context and multilingual support.

To run Mistral Nemo 12B Instruct locally: Q5_K_M ~8-9GB — fits on 12GB GPUs.

Apache 2.0 with strong multilingual capabilities — good balance of size and quality.

VRAM at each quantization

Assumes 8k context. KV cache grows linearly with context length.

QuantWeightsKV cacheTotal
FP3248.8 GB1.34 GB56.2 GB
BF1624.4 GB1.34 GB28.8 GB
FP1624.4 GB1.34 GB28.8 GB
Q8_012.2 GB1.34 GB15.2 GB
Q6_K10.0 GB1.34 GB12.7 GB
Q5_K_Mrec7.9 GB1.34 GB10.3 GB
Q4_K_M6.9 GB1.34 GB9.2 GB
Q3_K_M5.3 GB1.34 GB7.4 GB
Q2_K4.0 GB1.34 GB6.0 GB
NVFP4cuda6.1 GB1.34 GB8.3 GB

KV cache shown at 8k context (FP16). NVFP4 requires a CUDA GPU. Enable TurboQuant in the calculator to see reduced KV cache estimates.

Benchmarks

MMLU-Pro
35.6
IFEval
63.8
MATH
55

GPUs that run Mistral Nemo 12B Instruct natively (102)

Plus 5 GPUs that run it with CPU offload (slower)
Hugging Face ↗Ollama ↗Released 2024-07-18

Compare Mistral Nemo 12B Instruct with other models

Frequently asked questions

What are the VRAM requirements for Mistral Nemo 12B Instruct?
Mistral Nemo 12B Instruct requires approximately 9.2 GB of VRAM at Q4_K_M quantization, 15.2 GB at Q8, and 28.8 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
How many parameters does Mistral Nemo 12B Instruct have?
Mistral Nemo 12B Instruct has 12.2 billion parameters.
How capable is Mistral Nemo 12B Instruct?
Mistral Nemo 12B Instruct has an MMLU-Pro score of 35.6, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
Can Mistral Nemo 12B Instruct run on a 16 GB GPU?
Yes. Mistral Nemo 12B Instruct needs 9.2 GB at Q4_K_M, which fits in a 16 GB GPU like the RTX 4080 or RTX 4070 Ti Super.
What is the smallest quantization for Mistral Nemo 12B Instruct that fits in 24 GB of VRAM?
At NVFP4, Mistral Nemo 12B Instruct needs 8.3 GB — the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Mistral Nemo 12B Instruct locally?
A 16 GB GPU is enough. At Q4_K_M, Mistral Nemo 12B Instruct needs 9.2 GB VRAM. Good options: RTX 4080 (16 GB), RTX 4070 Ti Super (16 GB).