SmolLM2 360M Instruct
SmolLM2 360M Instruct needs roughly 0.6 GB VRAM at Q4_K_M quantization (1.2 GB at FP16). 117 GPUs we track can run it fully in VRAM at 8k context.
117 GPUs run this natively · 1 with CPU offload
Hugging Face0.36B params8k contextApache 2.0Commercial use ok
SmolLM2 360M Instruct is a 0.36B parameter dense model developed by Hugging Face. Ultra-compact 360M model for mobile and embedded.
To run SmolLM2 360M Instruct locally: Q8_K_M ~500MB and runs on virtually any device.
360M parameters, smallest practical LLM for on-device use.
VRAM at each quantization
Numbers here are computed at 8k context. Because KV cache grows linearly with context length, expect higher totals at longer sequence lengths.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP32 | 1.4 GB | 0.34 GB | 2.0 GB |
| BF16 | 0.7 GB | 0.34 GB | 1.2 GB |
| FP16 | 0.7 GB | 0.34 GB | 1.2 GB |
| Q8_0rec | 0.4 GB | 0.34 GB | 0.8 GB |
| Q6_K | 0.3 GB | 0.34 GB | 0.7 GB |
| Q5_K_M | 0.3 GB | 0.34 GB | 0.7 GB |
| Q4_K_M | 0.2 GB | 0.34 GB | 0.6 GB |
| Q3_K_M | 0.2 GB | 0.34 GB | 0.6 GB |
| Q2_K | 0.1 GB | 0.34 GB | 0.5 GB |
| NVFP4cuda | 0.2 GB | 0.34 GB | 0.6 GB |
Shown at 8k context with FP16 KV cache. NVFP4 needs a CUDA GPU to run. Toggle TurboQuant in the calculator to view compressed KV cache numbers.
Benchmarks
GPUs that run SmolLM2 360M Instruct natively (117)
- NVIDIA RTX 5090BF16 · 386.2 t/s
- NVIDIA RTX 5080BF16 · 315.5 t/s
- NVIDIA RTX 5070 TiBF16 · 306.9 t/s
- NVIDIA RTX 5070BF16 · 269.9 t/s
- NVIDIA RTX 5060 Ti 16GBBF16 · 217.5 t/s
Show 112 more
- NVIDIA RTX 5060 Ti 8GBBF16 · 217.5 t/s
- NVIDIA RTX 5060BF16 · 217.5 t/s
- NVIDIA RTX 5050BF16 · 176.4 t/s
- NVIDIA RTX 4090BF16 · 321.5 t/s
- NVIDIA RTX 4080BF16 · 278.3 t/s
- NVIDIA RTX 4070 Ti SUPERBF16 · 269.9 t/s
- NVIDIA RTX 4070 TiBF16 · 232.6 t/s
- NVIDIA RTX 4070 SUPERBF16 · 232.6 t/s
- NVIDIA RTX 4070BF16 · 232.6 t/s
- NVIDIA RTX 4060 Ti 16GBBF16 · 164.3 t/s
- NVIDIA RTX 4060BF16 · 158 t/s
- NVIDIA RTX 3090BF16 · 312.4 t/s
- NVIDIA RTX 3090 TiBF16 · 321.5 t/s
- NVIDIA RTX 3080 10GBBF16 · 285.9 t/s
- NVIDIA RTX 3060 12GBBF16 · 190.4 t/s
- NVIDIA B300 288GBBF16 · 403.9 t/s
- NVIDIA B200 180GBBF16 · 403.9 t/s
- NVIDIA H200 141GBBF16 · 388.2 t/s
- NVIDIA H100 80GBBF16 · 372.6 t/s
- NVIDIA A100 80GBBF16 · 343.1 t/s
- NVIDIA A100 40GBBF16 · 322.9 t/s
- NVIDIA L40SBF16 · 269.2 t/s
- NVIDIA RTX A6000BF16 · 287.2 t/s
- NVIDIA RTX 4000 AdaBF16 · 176.4 t/s
- NVIDIA RTX 4500 AdaBF16 · 212.9 t/s
- NVIDIA RTX 5000 AdaBF16 · 249.8 t/s
- NVIDIA RTX 6000 AdaBF16 · 315.5 t/s
- NVIDIA RTX Pro 6000BF16 · 386.2 t/s
- NVIDIA DGX Spark (128GB)BF16 · 158.4 t/s
- AMD Radeon RX 9070 XTBF16 · 263.6 t/s
- AMD Radeon RX 7900 XTXBF16 · 315.5 t/s
- AMD Radeon RX 7900 XTBF16 · 292.5 t/s
- AMD Radeon RX 7900 GREBF16 · 249.8 t/s
- AMD Radeon RX 7800 XTBF16 · 260.3 t/s
- AMD Radeon RX 6800 XTBF16 · 234.6 t/s
- AMD Radeon PRO W7800BF16 · 249.8 t/s
- AMD Radeon PRO W7900BF16 · 302.3 t/s
- AMD Instinct MI300XBF16 · 391.8 t/s
- AMD Radeon AI PRO R9700 32GBBF16 · 263.6 t/s
- AMD Strix Halo (128GB)BF16 · 151.4 t/s
- AMD Strix Halo (96GB)BF16 · 151.4 t/s
- AMD Strix Halo (64GB)BF16 · 151.4 t/s
- AMD Strix Halo (32GB)BF16 · 151.4 t/s
- Apple M5 Ultra (512GB)BF16 · 909.5 t/s
- Apple M5 Ultra (256GB)BF16 · 909.5 t/s
- Apple M5 Ultra (96GB)BF16 · 909.5 t/s
- Apple M5 Max (128GB)BF16 · 465.4 t/s
- Apple M5 Max (64GB)BF16 · 465.4 t/s
- Apple M5 Max (48GB)BF16 · 465.4 t/s
- Apple M5 Max (36GB)BF16 · 348.6 t/s
- Apple M5 Pro (64GB)BF16 · 232.7 t/s
- Apple M5 Pro (48GB)BF16 · 232.7 t/s
- Apple M5 Pro (24GB)BF16 · 232.7 t/s
- Apple M5 (32GB)BF16 · 116 t/s
- Apple M5 (16GB)BF16 · 116 t/s
- Apple M6 (32GB)BF16 · 128.8 t/s
- Apple M6 (16GB)BF16 · 128.8 t/s
- Apple M4 Max (128GB)BF16 · 413.8 t/s
- Apple M4 Max (64GB)BF16 · 413.8 t/s
- Apple M4 Max (48GB)BF16 · 413.8 t/s
- Apple M4 Max (36GB)BF16 · 310.7 t/s
- Apple M4 Pro (48GB)BF16 · 206.9 t/s
- Apple M4 Pro (24GB)BF16 · 206.9 t/s
- Apple M4 (32GB)BF16 · 90.9 t/s
- Apple M4 (16GB)BF16 · 90.9 t/s
- Apple M3 Ultra (512GB)BF16 · 620.7 t/s
- Apple M3 Ultra (256GB)BF16 · 620.7 t/s
- Apple M3 Ultra (96GB)BF16 · 620.7 t/s
- Apple M3 Max (128GB)BF16 · 303.2 t/s
- Apple M3 Max (96GB)BF16 · 227.4 t/s
- Apple M3 Max (64GB)BF16 · 303.2 t/s
- Apple M3 Max (48GB)BF16 · 303.2 t/s
- Apple M3 Max (36GB)BF16 · 227.4 t/s
- Apple M3 Pro (36GB)BF16 · 113.7 t/s
- Apple M3 Pro (18GB)BF16 · 113.7 t/s
- Apple M3 (24GB)BF16 · 75.8 t/s
- Apple M3 (16GB)BF16 · 75.8 t/s
- Apple M2 Ultra (192GB)BF16 · 606.3 t/s
- Apple M2 Ultra (64GB)BF16 · 606.3 t/s
- Apple M2 Max (96GB)BF16 · 303.2 t/s
- Apple M2 Max (64GB)BF16 · 303.2 t/s
- Apple M2 Max (32GB)BF16 · 303.2 t/s
- Apple M2 Pro (32GB)BF16 · 151.6 t/s
- Apple M2 Pro (16GB)BF16 · 151.6 t/s
- Apple M2 (24GB)BF16 · 75.8 t/s
- Apple M2 (16GB)BF16 · 75.8 t/s
- Apple M1 Ultra (128GB)BF16 · 606.3 t/s
- Apple M1 Ultra (64GB)BF16 · 606.3 t/s
- Apple M1 Max (64GB)BF16 · 303.2 t/s
- Apple M1 Max (32GB)BF16 · 303.2 t/s
- Apple M1 Pro (32GB)BF16 · 151.6 t/s
- Apple M1 Pro (16GB)BF16 · 151.6 t/s
- Apple M1 (16GB)BF16 · 51.5 t/s
- Intel Arc B580 12GBBF16 · 219.8 t/s
- Intel Arc B570 10GBBF16 · 197 t/s
- Intel Arc Pro B70 32GBBF16 · 256.9 t/s
- Intel Arc Pro B60 24GBBF16 · 197 t/s
- Intel Arc Pro B50 16GBBF16 · 137.5 t/s
- Intel Arc A770 16GBBF16 · 246.2 t/s
- Intel Arc A770 8GBBF16 · 234.6 t/s
- Intel Arc A750 8GBBF16 · 234.6 t/s
- Intel Arc A580 8GBBF16 · 234.6 t/s
- Intel Arc A380 6GBBF16 · 119.5 t/s
- Intel Arc A310 4GBBF16 · 86.3 t/s
- Intel Arc Pro A60 12GBBF16 · 198.3 t/s
- Intel Arc Pro A50 6GBBF16 · 122.4 t/s
- Intel Arc Pro A40 6GBBF16 · 122.4 t/s
- Intel Data Center GPU Max 1550BF16 · 371.5 t/s
- Intel Data Center GPU Max 1100BF16 · 302.9 t/s
- Intel Arc 140V (32GB)BF16 · 93.7 t/s
- Intel Arc 140V (16GB)BF16 · 93.7 t/s
- Intel Arc 130V (16GB)BF16 · 93.7 t/s
Plus 1 GPU that runs it with CPU offload (slower)
- CPU only (system RAM)BF16 · 37.9 t/s
No GPU? Rent an RTX 3080 10GB from $0.17/hr ↗
Prices as of Oct 17, 2026. Rates change; check RunPod for current pricing. Affiliate link: CanItRun may earn a commission. Doesn't affect the fit calculation above.
Frequently asked questions
- What are the VRAM requirements for SmolLM2 360M Instruct?
- SmolLM2 360M Instruct requires approximately 0.6 GB of VRAM at Q4_K_M quantization, 0.8 GB at Q8, and 1.2 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
- How many parameters does SmolLM2 360M Instruct have?
- SmolLM2 360M Instruct has 0.36 billion parameters.
- How capable is SmolLM2 360M Instruct?
- SmolLM2 360M Instruct has an MMLU-Pro score of 8, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
- Can SmolLM2 360M Instruct run on a 16 GB GPU?
- Yes. SmolLM2 360M Instruct needs 0.6 GB at Q4_K_M, which fits in a 16 GB GPU like the RTX 4080 or RTX 5070 Ti.
- What is the smallest quantization for SmolLM2 360M Instruct that fits in 24 GB of VRAM?
- At BF16, SmolLM2 360M Instruct needs 1.2 GB, the highest-quality quantization that fits in 24 GB of VRAM.
- What GPU do I need to run SmolLM2 360M Instruct locally?
- A 16 GB GPU is enough. At Q4_K_M, SmolLM2 360M Instruct needs 0.6 GB VRAM. Good options: RTX 4080 (16 GB), RTX 5070 Ti (16 GB).
- Can I run SmolLM2 360M Instruct in the cloud?
- Yes. The cheapest RunPod setup we found that holds SmolLM2 360M Instruct entirely in GPU memory at 8k context is 1× RTX 3080 10GB at BF16, at roughly 285.9 tokens per second and from $0.17 per hour for the whole pod. Prices as of Oct 17, 2026; RunPod rates change, so check their pricing page for the current figure.