Can I run a 13B model on 8 GB VRAM?

In theory, a 13B model at Q2_K might fit (roughly 5.5 GB for weights), but Q2_K quantization degrades quality enough that you are often better off running a 7B model at Q4_K_M or Q5_K_M. The 7B at higher quant will likely produce better output than the 13B at Q2_K. If you need 13B-class performance, upgrading to 12 or 16 GB VRAM is the more practical path.

Is 8 GB VRAM still useful for LLMs in 2026?

Yes, absolutely. The 7B model class has improved dramatically. A 2026 7B model like Qwen 2.5 7B or Llama 3.1 8B outperforms the original GPT-3 (175B) on many benchmarks. For personal assistants, code completion, summarization, and lightweight chat, 8 GB is perfectly functional. It only becomes limiting when you need long context, large models, or multimodal capabilities.

Should I buy an 8 GB or 12 GB GPU for local LLMs?

If you are buying new hardware specifically for local LLMs, 12 GB is the better investment. The RTX 3060 12GB can often be found at similar prices to 8 GB cards on the used market, and that extra 4 GB opens up Q8_0 quantization for 7B models and tight-fit Q4_K_M for some 14B models. The 50% more VRAM makes a meaningful difference in daily usability.

Which inference engine is best for 8 GB cards?

Ollama is the easiest to set up and handles VRAM management automatically. For maximum control, llama.cpp lets you specify exact GPU layer counts and context sizes. On AMD GPUs, make sure you are using a backend with GPU acceleration (ROCm on Linux, Vulkan on Windows) rather than CPU-only inference, which is dramatically slower. LM Studio provides a GUI if you prefer not to use the command line.

Best LLMs You Can Run on 8 GB VRAM (2026)

CanItRunJune 28, 20268 min readVRAM Guides

What 8 GB of VRAM Actually Gets You

Eight gigabytes of VRAM puts you in the domain of models with roughly 1 to 8 billion parameters at practical quantization levels. This is not a limitation to dismiss: 7B-class models in 2026 are dramatically more capable than the 175B models of just a few years ago, thanks to improvements in training data quality, architecture, and alignment. At Q4_K_M quantization, a 7B model consumes approximately 4.5 GB for weights, leaving about 3 GB for the KV cache, runtime overhead, and whatever else your GPU is doing. That is enough headroom for roughly 4096-8192 tokens of context, depending on the model's architecture. You will not be running 14B or larger models on 8 GB in any practical sense. Even at the lowest usable quantization (Q2_K), a 14B model needs about 6 GB for weights alone, leaving almost no room for inference. Accept this constraint and focus on finding the best 7B-class model for your specific use case rather than trying to squeeze a bigger model into too little memory.

Best for General Use: Llama 3.1 8B

Llama 3.1 8B remains one of the strongest all-around models in the 8 GB VRAM tier. It handles conversation, summarization, creative writing, and light analysis tasks well. At Q4_K_M, the weights occupy about 5 GB, fitting comfortably on any 8 GB card with room for 4K context. Instruction following is solid, and the model benefits from Meta's extensive RLHF tuning. The ecosystem support is unmatched: every major inference engine, every GUI frontend, and every API wrapper supports Llama 3.1 8B out of the box. If you are unsure what model to start with on 8 GB hardware, this is the safe default. Its weakness is specialized tasks: dedicated coding or reasoning models outperform it in those specific domains, but for a general-purpose assistant, it is hard to beat at this size.

# Install and run with Ollama
ollama run llama3.1:8b

# Check VRAM usage after loading
nvidia-smi

Best for Coding: Qwen 2.5 Coder 7B

If your primary use case is code generation, completion, or debugging, Qwen 2.5 Coder 7B is the strongest option in the 8 GB tier. Alibaba trained it on a massive code-specific dataset, and it outperforms Llama 3.1 8B on coding benchmarks by a significant margin. It handles Python, JavaScript, TypeScript, Rust, Go, and most other popular languages well. At Q4_K_M, it fits in the same envelope as other 7B models: roughly 4.5 GB for weights. The instruct variant is tuned for following coding instructions, explaining code, and generating implementations from natural language descriptions. One practical note: Qwen 2.5 Coder uses a large vocabulary (151K tokens), which means the embedding layer is proportionally larger. This does not meaningfully affect VRAM at Q4_K_M, but it does mean the Q8_0 version is slightly larger than other 7B models and may be tight on 8 GB cards with short context. For pure coding work on a budget GPU, this is the model to install first.

# Run Qwen 2.5 Coder 7B
ollama run qwen2.5-coder:7b

# Use with continue.dev or other IDE integrations
# Set the model to qwen2.5-coder:7b in your IDE extension settings

Best for Reasoning: Phi-3.5 Mini and Gemma 3 4B

For users who want reasoning capability on 8 GB VRAM, two models stand out. Phi-3.5 Mini (3.8B parameters) fits easily at Q8_0 with generous context headroom. It punches well above its parameter count on math and logic benchmarks thanks to Microsoft's synthetic data training approach. It will not match a 70B model on complex multi-step reasoning, but for straightforward math, logic puzzles, and structured analysis, it is surprisingly good. At Q4_K_M it uses only about 2.5 GB of VRAM, leaving substantial headroom for long context windows. Gemma 3 4B is another strong option in this space: Google's training approach gives it solid performance on reasoning benchmarks relative to its size, and at Q4_K_M it uses only about 2.8 GB of VRAM. Both models are small enough that you can comfortably run them at Q8_0 on an 8 GB card, which is rare for this VRAM tier and means you get near-lossless quality. If reasoning is your primary need but you are budget-constrained on hardware, these sub-4B models at high quantization are better than trying to squeeze a 7B model at the lowest possible quant.

# Run Phi-3.5 Mini for reasoning tasks
ollama run phi3.5:3.8b

# Run Gemma 3 4B as an alternative
ollama run gemma3:4b

Quantization Trade-offs at 8 GB

At the 8 GB tier, your quantization choice meaningfully affects both quality and usability. Q4_K_M is the default recommendation for 7B models because it preserves the vast majority of model quality while leaving enough VRAM for practical context lengths. Going to Q3_K_M saves roughly 1 GB but introduces noticeable degradation: you will see more grammatical errors, less coherent long-form output, and weaker performance on knowledge-intensive tasks. Q2_K is generally not recommended for 7B models because the quality loss is substantial. Going the other direction, Q5_K_M adds roughly 0.8 GB over Q4_K_M and gives a small but real quality improvement. If you have exactly 8 GB and want the best quality, use Q4_K_M on a 7B model. If you have 10-12 GB, step up to Q5_K_M or even Q8_0. One important caveat: not all 7B models are created equal. Some models have larger embedding tables or different architectures that shift these numbers by a few hundred megabytes. Always check the actual file size of the quantized GGUF before assuming it will fit.

Context Length Limitations on 8 GB

Context length is where 8 GB VRAM really feels constraining. After loading a 7B model at Q4_K_M (approximately 4.5 GB), you have roughly 3 GB left for the KV cache and runtime overhead. The runtime overhead consumes about 500 MB, leaving 2.5 GB for the KV cache. For a model like Llama 3.1 8B with grouped-query attention, this translates to roughly 8K-12K tokens of context at FP16 KV cache precision. If you enable KV cache quantization (supported in recent versions of llama.cpp and Ollama), you can stretch this to 16K-24K tokens. The practical impact: you can have a moderately long conversation, process a short document, or work with a few files of code. You cannot process a full novel, a long technical specification, or maintain context across an extended multi-turn session without the model forgetting earlier parts. If you frequently need long context, upgrading to a 12 or 16 GB card is a better investment than trying to optimize around the 8 GB limit.

Practical Setup Tips for 8 GB Cards

To get the most out of an 8 GB GPU for local LLMs, start with a few practical optimizations. First, close your browser or disable GPU acceleration in browser settings. Chrome and Firefox can consume 500 MB to 2 GB of VRAM for hardware-accelerated rendering, eating directly into your LLM headroom. Second, disable any overlays (Discord, Steam, GeForce Experience) that consume VRAM. Third, set your desktop to a single monitor if you are running multi-display. Fourth, on Linux, consider using a lightweight desktop environment or even a TTY session for inference to minimize background VRAM usage. Ollama is the easiest way to get started: it handles model downloading, quantization selection, and GPU management automatically. For more control, llama.cpp lets you specify exact VRAM budgets with the -ngl (layers to GPU) flag. If a model does not quite fit, try reducing -ngl by 5-10 layers instead of switching to a smaller model. Partial offloading to CPU is slower, but the layers that remain on GPU still benefit from GPU acceleration. On AMD cards, make sure ROCm is properly configured (Linux) or use the Vulkan backend (Windows) for GPU acceleration.

# Check current VRAM usage before running a model
nvidia-smi

# Kill GPU-hungry background processes
sudo fuser -v /dev/nvidia*

# Run with explicit layer control in llama.cpp
./llama-cli -m model-Q4_K_M.gguf -ngl 33 -c 4096

Models to Skip on 8 GB VRAM

Not every model marketed as small actually fits on 8 GB. Models with 12-14B parameters, even at Q3_K_M, typically need 7-9 GB for weights alone, leaving no room for inference. Mixture-of-experts models like Mixtral 8x7B have 46.7B total parameters (12.9B active), and their weight files are sized by total parameters, not active ones, so they need 24+ GB even at Q4_K_M. Any model with a 128K default context length will try to allocate a large KV cache on startup, which can cause out-of-memory errors before inference even begins. Reduce the context length explicitly. Also be cautious with vision-language models at this tier: the image encoder adds substantial overhead beyond the text model weights. If you want multimodal capability on 8 GB, Gemma 3 4B with its vision mode is one of the few that fits, though you will have very limited context for the text portion during image analysis.

Frequently asked questions

Can I run a 13B model on 8 GB VRAM?: In theory, a 13B model at Q2_K might fit (roughly 5.5 GB for weights), but Q2_K quantization degrades quality enough that you are often better off running a 7B model at Q4_K_M or Q5_K_M. The 7B at higher quant will likely produce better output than the 13B at Q2_K. If you need 13B-class performance, upgrading to 12 or 16 GB VRAM is the more practical path.
Is 8 GB VRAM still useful for LLMs in 2026?: Yes, absolutely. The 7B model class has improved dramatically. A 2026 7B model like Qwen 2.5 7B or Llama 3.1 8B outperforms the original GPT-3 (175B) on many benchmarks. For personal assistants, code completion, summarization, and lightweight chat, 8 GB is perfectly functional. It only becomes limiting when you need long context, large models, or multimodal capabilities.
Should I buy an 8 GB or 12 GB GPU for local LLMs?: If you are buying new hardware specifically for local LLMs, 12 GB is the better investment. The RTX 3060 12GB can often be found at similar prices to 8 GB cards on the used market, and that extra 4 GB opens up Q8_0 quantization for 7B models and tight-fit Q4_K_M for some 14B models. The 50% more VRAM makes a meaningful difference in daily usability.
Which inference engine is best for 8 GB cards?: Ollama is the easiest to set up and handles VRAM management automatically. For maximum control, llama.cpp lets you specify exact GPU layer counts and context sizes. On AMD GPUs, make sure you are using a backend with GPU acceleration (ROCm on Linux, Vulkan on Windows) rather than CPU-only inference, which is dramatically slower. LM Studio provides a GUI if you prefer not to use the command line.