GLM-4.6V 106B
GLM-4.6V 106B needs roughly 61.1GB VRAM at Q4_K_M quantization (239.2GB at FP16). 29 GPUs we track can run it fully in VRAM at 8k context.
Z.ai106B params12B active (MoE)128k contextMITCommercial use ok
VRAM at each quantization
Assumes 8k context. KV cache grows linearly with context length.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP16 | 212.0 GB | 1.54 GB | 239.2 GB |
| Q8 | 106.0 GB | 1.54 GB | 120.5 GB |
| Q6_K | 79.5 GB | 1.54 GB | 90.8 GB |
| Q5_K_M | 66.3 GB | 1.54 GB | 75.9 GB |
| Q4_K_M | 53.0 GB | 1.54 GB | 61.1 GB |
| Q3_K_M | 42.4 GB | 1.54 GB | 49.2 GB |
| Q2_K | 31.8 GB | 1.54 GB | 37.3 GB |
Benchmarks
Benchmarks for this model are not yet available on the Open LLM Leaderboard v2. This is common for recently released models. Check back soon.
GPUs that run GLM-4.6V 106B natively (29)
- NVIDIA H100 80GBQ5_K_M · 491.3 t/s
- NVIDIA A100 80GBQ5_K_M · 299.1 t/s
- NVIDIA A100 40GBQ2_K · 475.1 t/s
- NVIDIA L40SQ2_K · 264 t/s
- NVIDIA RTX A6000Q2_K · 234.7 t/s
- NVIDIA RTX 6000 AdaQ2_K · 293.3 t/s
- NVIDIA DGX Spark (128GB)Q8 · 25 t/s
- AMD Instinct MI300XQ8 · 485.8 t/s
- AMD Strix Halo (128GB)Q8 · 23.5 t/s
- AMD Strix Halo (96GB)Q6_K · 31.3 t/s
- AMD Strix Halo (64GB)Q3_K_M · 58.7 t/s
- Apple M4 Ultra (384GB)FP16 · 50.1 t/s
- Apple M4 Ultra (192GB)Q8 · 100.1 t/s
- Apple M4 Max (128GB)Q8 · 50.1 t/s
- Apple M4 Max (96GB)Q6_K · 66.7 t/s
- Apple M4 Max (64GB)Q3_K_M · 125.1 t/s
- Apple M4 Max (48GB)Q2_K · 166.8 t/s
- Apple M4 Pro (48GB)Q2_K · 83.4 t/s
- Apple M3 Max (128GB)Q8 · 36.7 t/s
- Apple M3 Max (96GB)Q6_K · 48.9 t/s
- Apple M3 Max (64GB)Q3_K_M · 91.7 t/s
- Apple M3 Max (48GB)Q2_K · 122.2 t/s
- Apple M2 Ultra (384GB)FP16 · 36.7 t/s
- Apple M2 Ultra (192GB)Q8 · 73.3 t/s
- Apple M2 Max (96GB)Q6_K · 48.9 t/s
- Apple M2 Max (64GB)Q3_K_M · 91.7 t/s
- Apple M1 Ultra (128GB)Q8 · 73.3 t/s
- Apple M1 Ultra (64GB)Q3_K_M · 183.3 t/s
- Apple M1 Max (64GB)Q3_K_M · 91.7 t/s
Plus 13 GPUs that run it with CPU offload (slower)
- NVIDIA RTX 5090Q3_K_M · 93.3 t/s
- NVIDIA RTX 4090Q2_K · 70 t/s
- NVIDIA RTX 4080Q2_K · 49.8 t/s
- NVIDIA RTX 4070 TiQ2_K · 35 t/s
- NVIDIA RTX 4070Q2_K · 35 t/s
- NVIDIA RTX 4060 Ti 16GBQ2_K · 20 t/s
- NVIDIA RTX 3090Q2_K · 65 t/s
- NVIDIA RTX 3090 TiQ2_K · 70 t/s
- NVIDIA RTX 3060 12GBQ2_K · 25 t/s
- AMD Radeon RX 7900 XTXQ2_K · 66.7 t/s
- AMD Radeon RX 7900 XTQ2_K · 55.6 t/s
- AMD Radeon RX 6800 XTQ2_K · 35.6 t/s
- Intel Arc A770 16GBQ2_K · 38.9 t/s
Notes
Multimodal vision-language variant of GLM-4.6 Air — supports image and video inputs.