CanItRun Logocanitrun.

Gemma 2 9B Instruct

Gemma 2 9B Instruct needs roughly 9.4 GB VRAM at Q4_K_M quantization (23.8 GB at FP16). 111 GPUs we track can run it fully in VRAM at 8k context.

111 GPUs run this natively · 5 with CPU offload

Google9.2B params8k contextGemmaCommercial use ok

Gemma 2 9B Instruct is a 9.2B parameter dense model developed by Google. June 2024 9B model with knowledge distillation from 27B teacher, best performance for its size class.

To run Gemma 2 9B Instruct locally: Q5_K_M ~6-7GB and runs on 8GB GPUs. Excellent quality-per-VRAM ratio.

MMLU-Pro 32.0%, competitive with models 2-3× larger. Trained 50× beyond compute-optimal.

VRAM at each quantization

The table below assumes 8k context. KV cache size scales linearly with how much context you use.

QuantWeightsKV cacheTotal
FP3236.8 GB2.82 GB44.4 GB
BF1618.4 GB2.82 GB23.8 GB
FP1618.4 GB2.82 GB23.8 GB
Q8_09.8 GB2.82 GB14.1 GB
Q6_K7.5 GB2.82 GB11.6 GB
Q5_K_Mrec6.5 GB2.82 GB10.5 GB
Q4_K_M5.6 GB2.82 GB9.4 GB
Q3_K_M4.4 GB2.82 GB8.1 GB
Q2_K3.5 GB2.82 GB7.1 GB
NVFP4cuda4.6 GB2.82 GB8.3 GB

KV cache figures assume 8k context at FP16. NVFP4 quantization requires a CUDA-capable GPU. Enable TurboQuant in the calculator to see reduced KV cache estimates.

Benchmarks

MMLU-Pro
32.0
GPQA Diamond
31.5
IFEval
74.4
MATH
44.3
Arena ELO
1190.0

GPUs that run Gemma 2 9B Instruct natively (111)

Show 106 more
Plus 5 GPUs that run it with CPU offload (slower)
Hugging Face ↗Ollama ↗Released 2024-06-27

Compare Gemma 2 9B Instruct with other models

Explore

Browse all guides →Browse all models →Browse all GPUs →

Frequently asked questions

What are the VRAM requirements for Gemma 2 9B Instruct?
Gemma 2 9B Instruct requires approximately 9.4 GB of VRAM at Q4_K_M quantization, 14.1 GB at Q8, and 23.8 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
How many parameters does Gemma 2 9B Instruct have?
Gemma 2 9B Instruct has 9.2 billion parameters.
How capable is Gemma 2 9B Instruct?
Gemma 2 9B Instruct has an MMLU-Pro score of 32, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
Can Gemma 2 9B Instruct run on a 16 GB GPU?
Yes. Gemma 2 9B Instruct needs 9.4 GB at Q4_K_M, which fits in a 16 GB GPU like the RTX 4080 or RTX 5070 Ti.
What is the smallest quantization for Gemma 2 9B Instruct that fits in 24 GB of VRAM?
At BF16, Gemma 2 9B Instruct needs 23.8 GB, the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Gemma 2 9B Instruct locally?
A 16 GB GPU is enough. At Q4_K_M, Gemma 2 9B Instruct needs 9.4 GB VRAM. Good options: RTX 4080 (16 GB), RTX 5070 Ti (16 GB).