Llama 3.1 70B Instruct
Llama 3.1 70B Instruct needs roughly 35.0GB VRAM at Q4 quantization (140.0GB at FP16). 33 GPUs we track can run it fully in VRAM at 8k context.
Meta70B params125k contextLlama 3.1 CommunityCommercial use ok
VRAM at each quantization
Assumes 8k context. KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP16 | 140.0 GB | 2.68 GB | 159.8 GB |
| Q8 | 70.0 GB | 2.68 GB | 81.4 GB |
| Q6_K | 52.5 GB | 2.68 GB | 61.8 GB |
| Q5_K_M | 43.8 GB | 2.68 GB | 52.0 GB |
| Q4_K_M | 35.0 GB | 2.68 GB | 42.2 GB |
| Q3_K_M | 28.0 GB | 2.68 GB | 34.4 GB |
| Q2_K | 21.0 GB | 2.68 GB | 26.5 GB |
Benchmarks
GPUs that run Llama 3.1 70B Instruct natively (33)
- NVIDIA RTX 5090Q2_K · 85.3 t/s
- NVIDIA H100 80GBQ6_K · 63.8 t/s
- NVIDIA A100 80GBQ6_K · 38.8 t/s
- NVIDIA A100 40GBQ3_K_M · 55.5 t/s
- NVIDIA L40SQ4_K_M · 24.7 t/s
- NVIDIA RTX A6000Q4_K_M · 21.9 t/s
- NVIDIA RTX 6000 AdaQ4_K_M · 27.4 t/s
- AMD Instinct MI300XFP16 · 37.9 t/s
- Apple M4 Ultra (384GB)FP16 · 7.8 t/s
- Apple M4 Ultra (192GB)FP16 · 7.8 t/s
- Apple M4 Max (128GB)Q8 · 7.8 t/s
- Apple M4 Max (96GB)Q8 · 7.8 t/s
- Apple M4 Max (64GB)Q5_K_M · 12.5 t/s
- Apple M4 Max (48GB)Q4_K_M · 15.6 t/s
- Apple M4 Pro (48GB)Q4_K_M · 7.8 t/s
- Apple M4 (32GB)Q2_K · 5.7 t/s
- Apple M3 Max (128GB)Q8 · 5.7 t/s
- Apple M3 Max (96GB)Q8 · 5.7 t/s
- Apple M3 Max (64GB)Q5_K_M · 9.1 t/s
- Apple M3 Max (48GB)Q4_K_M · 11.4 t/s
- Apple M3 Max (36GB)Q2_K · 19 t/s
- Apple M3 Pro (36GB)Q2_K · 7.1 t/s
- Apple M2 Ultra (384GB)FP16 · 5.7 t/s
- Apple M2 Ultra (192GB)FP16 · 5.7 t/s
- Apple M2 Max (96GB)Q8 · 5.7 t/s
- Apple M2 Max (64GB)Q5_K_M · 9.1 t/s
- Apple M2 Max (32GB)Q2_K · 19 t/s
- Apple M2 Pro (32GB)Q2_K · 9.5 t/s
- Apple M1 Ultra (128GB)Q8 · 11.4 t/s
- Apple M1 Ultra (64GB)Q5_K_M · 18.3 t/s
- Apple M1 Max (64GB)Q5_K_M · 9.1 t/s
- Apple M1 Max (32GB)Q2_K · 19 t/s
- Apple M1 Pro (32GB)Q2_K · 9.5 t/s
Plus 15 GPUs that run it with CPU offload (slower)
- NVIDIA RTX 4090Q4_K_M · 7.2 t/s
- NVIDIA RTX 4080Q3_K_M · 6.4 t/s
- NVIDIA RTX 4070 TiQ3_K_M · 4.5 t/s
- NVIDIA RTX 4070Q3_K_M · 4.5 t/s
- NVIDIA RTX 4060 Ti 16GBQ3_K_M · 2.6 t/s
- NVIDIA RTX 4060Q2_K · 3.2 t/s
- NVIDIA RTX 3090Q4_K_M · 6.7 t/s
- NVIDIA RTX 3090 TiQ4_K_M · 7.2 t/s
- NVIDIA RTX 3080 10GBQ3_K_M · 6.8 t/s
- NVIDIA RTX 3060 12GBQ3_K_M · 3.2 t/s
- AMD Radeon RX 7900 XTXQ4_K_M · 6.9 t/s
- AMD Radeon RX 7900 XTQ4_K_M · 5.7 t/s
- AMD Radeon RX 6800 XTQ3_K_M · 4.6 t/s
- Intel Arc A770 16GBQ3_K_M · 5 t/s
- CPU only (system RAM)Q2_K · 0.5 t/s