Llama 3.2 3B Instruct
Llama 3.2 3B Instruct needs roughly 1.6GB VRAM at Q4 quantization (6.4GB at FP16). 60 GPUs we track can run it fully in VRAM at 8k context.
Meta3.2B params125k contextLlama 3.2 CommunityCommercial use ok
VRAM at each quantization
Assumes 8k context. KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP16 | 6.4 GB | 0.94 GB | 8.2 GB |
| Q8 | 3.2 GB | 0.94 GB | 4.6 GB |
| Q6_K | 2.4 GB | 0.94 GB | 3.7 GB |
| Q5_K_M | 2.0 GB | 0.94 GB | 3.3 GB |
| Q4_K_M | 1.6 GB | 0.94 GB | 2.8 GB |
| Q3_K_M | 1.3 GB | 0.94 GB | 2.5 GB |
| Q2_K | 1.0 GB | 0.94 GB | 2.1 GB |
Benchmarks
GPUs that run Llama 3.2 3B Instruct natively (60)
- NVIDIA RTX 5090FP16 · 280 t/s
- NVIDIA RTX 4090FP16 · 157.5 t/s
- NVIDIA RTX 4080FP16 · 112 t/s
- NVIDIA RTX 4070 TiFP16 · 78.8 t/s
- NVIDIA RTX 4070FP16 · 78.8 t/s
- NVIDIA RTX 4060 Ti 16GBFP16 · 45 t/s
- NVIDIA RTX 4060Q8 · 85 t/s
- NVIDIA RTX 3090FP16 · 146.3 t/s
- NVIDIA RTX 3090 TiFP16 · 157.5 t/s
- NVIDIA RTX 3080 10GBFP16 · 118.8 t/s
- NVIDIA RTX 3060 12GBFP16 · 56.3 t/s
- NVIDIA H100 80GBFP16 · 523.4 t/s
- NVIDIA A100 80GBFP16 · 318.6 t/s
- NVIDIA A100 40GBFP16 · 243 t/s
- NVIDIA L40SFP16 · 135 t/s
- NVIDIA RTX A6000FP16 · 120 t/s
- NVIDIA RTX 6000 AdaFP16 · 150 t/s
- AMD Radeon RX 7900 XTXFP16 · 150 t/s
- AMD Radeon RX 7900 XTFP16 · 125 t/s
- AMD Radeon RX 6800 XTFP16 · 80 t/s
- AMD Instinct MI300XFP16 · 828.1 t/s
- Apple M4 Ultra (384GB)FP16 · 170.6 t/s
- Apple M4 Ultra (192GB)FP16 · 170.6 t/s
- Apple M4 Max (128GB)FP16 · 85.3 t/s
- Apple M4 Max (96GB)FP16 · 85.3 t/s
- Apple M4 Max (64GB)FP16 · 85.3 t/s
- Apple M4 Max (48GB)FP16 · 85.3 t/s
- Apple M4 Pro (48GB)FP16 · 42.7 t/s
- Apple M4 Pro (24GB)FP16 · 42.7 t/s
- Apple M4 (32GB)FP16 · 18.8 t/s
- Apple M4 (16GB)FP16 · 18.8 t/s
- Apple M3 Max (128GB)FP16 · 62.5 t/s
- Apple M3 Max (96GB)FP16 · 62.5 t/s
- Apple M3 Max (64GB)FP16 · 62.5 t/s
- Apple M3 Max (48GB)FP16 · 62.5 t/s
- Apple M3 Max (36GB)FP16 · 62.5 t/s
- Apple M3 Pro (36GB)FP16 · 23.4 t/s
- Apple M3 Pro (18GB)FP16 · 23.4 t/s
- Apple M3 (24GB)FP16 · 15.6 t/s
- Apple M3 (16GB)FP16 · 15.6 t/s
- Apple M3 (8GB)Q8 · 31.3 t/s
- Apple M2 Ultra (384GB)FP16 · 125 t/s
- Apple M2 Ultra (192GB)FP16 · 125 t/s
- Apple M2 Max (96GB)FP16 · 62.5 t/s
- Apple M2 Max (64GB)FP16 · 62.5 t/s
- Apple M2 Max (32GB)FP16 · 62.5 t/s
- Apple M2 Pro (32GB)FP16 · 31.3 t/s
- Apple M2 Pro (16GB)FP16 · 31.3 t/s
- Apple M2 (24GB)FP16 · 15.6 t/s
- Apple M2 (16GB)FP16 · 15.6 t/s
- Apple M2 (8GB)Q8 · 31.3 t/s
- Apple M1 Ultra (128GB)FP16 · 125 t/s
- Apple M1 Ultra (64GB)FP16 · 125 t/s
- Apple M1 Max (64GB)FP16 · 62.5 t/s
- Apple M1 Max (32GB)FP16 · 62.5 t/s
- Apple M1 Pro (32GB)FP16 · 31.3 t/s
- Apple M1 Pro (16GB)FP16 · 31.3 t/s
- Apple M1 (16GB)FP16 · 10.6 t/s
- Apple M1 (8GB)Q8 · 21.3 t/s
- Intel Arc A770 16GBFP16 · 87.5 t/s
Plus 1 GPUs that run it with CPU offload (slower)
- CPU only (system RAM)FP16 · 1.6 t/s
Notes
Small, efficient on-device model.