Can your GPU run this LLM?

Running large language models locally gives you privacy, control, and zero inference costs — but figuring out which models fit on your GPU is a manual, error-prone process. CanItRun eliminates the guesswork. We built this as a free, open tool for developers, researchers, and hobbyists who want to experiment with open-weight LLMs without cloud dependencies.

How it works:Model VRAM requirements are calculated from three components: base model weights (adjusted for quantization level), key-value cache for your target context length, and activation memory for inference. We then compare these requirements against a comprehensive database of real GPU specifications and community-reported benchmarks to tell you not just whether a model fits, but how it's likely to perform.

Who should use this:ML engineers prototyping locally, researchers on academic budgets, students learning about LLMs, and hobbyists running models on consumer hardware. If you're evaluating whether to upgrade your GPU or trying to squeeze the largest model onto your existing setup, this tool helps you make data-driven decisions.

What sets CanItRun apart: Unlike generic calculators, we maintain an extensive GPU database with real-world benchmarks, provide quantization-specific recommendations (from FP16 down to INT4), and show expected tokens-per-second performance based on community data. We also track emerging architectures and new GPU releases to keep recommendations current.

The landscape of large language models has shifted dramatically. What once required cloud APIs and expensive GPU clusters now runs on consumer hardware. Open-weight models like Llama 3, Mistral, and Phi have democratized access to powerful AI — but with hundreds of models ranging from 1B to 70B+ parameters, choosing the right one for your hardware remains challenging.

Key Trends Driving Local Inference

• Quantization breakthroughs: INT4 and Q4_K_M quantization now deliver near-FP16 quality at 4x smaller memory footprints, making 70B models runnable on dual 24GB GPUs.
• Architecture efficiency: MoE (Mixture of Experts) models like Mixtral and Grok-1 offer better throughput by activating only subsets of parameters per token.
• Consumer GPU advances: RTX 4090/5090 and Radeon RX 7900 XTX provide 24–48GB VRAM at consumer prices, while used enterprise cards (A100, A6000) remain popular for serious workloads.
• Inference frameworks: llama.cpp, vLLM, and MLX have matured dramatically, offering production-ready performance on Mac, Linux, and Windows.

Developers waste hours debugging OOM errors, manually calculating VRAM requirements, or downloading models only to discover they don't fit. CanItRun prevents these issues upfront:

• VRAM estimation errors: Raw parameter count × bytes per parameter ignores KV cache and activations — we account for all three components.
• Quantization confusion: Q4_0 vs Q4_K_M vs IQ4_XS have different size/speed tradeoffs. We show requirements for each quantization level.
• Context length surprises: A model that fits at 4K context may fail at 32K. We calculate KV cache growth based on your target sequence length.
• Performance guesswork: Fitting in VRAM doesn't guarantee usable throughput. We show community-reported tokens/second so you know what to expect.
• Hardware comparison: Evaluating GPU upgrades? Compare multiple cards side-by-side to see which gives the best model coverage for your budget.

Step 1: Choose Your Hardware

Select your GPU from the dropdown above. We support NVIDIA (RTX 30/40/50 series, Tesla, Quadro), AMD (RX 6000/7000, Instinct), Apple Silicon (M1–M4), and Intel Arc. For multi-GPU setups, select your primary card — we'll note when models require multiple GPUs.

Step 2: Explore Compatible Models

The calculator shows which models fit at each quantization level. Green "Yes" means the model fits comfortably. "Maybe" indicates tight margins where background processes could cause OOM. "No" means the model exceeds your VRAM even at the lowest quantization.

Step 3: Download and Run

Once you've identified compatible models, download GGUF files from Hugging Face and run them with llama.cpp, LM Studio, or Ollama. For best results:

• Use Q4_K_M or Q5_K_M quantization for the best quality/size balance
• Set -c (context) to match your use case — 4096 for chat, 8192+ for document analysis
• Enable GPU offloading with -ngl 99 to load all layers on your GPU
• Monitor VRAM with nvidia-smi or watch -n1 rocm-smi during first runs