Is NVIDIA or AMD better for running LLMs locally?

NVIDIA has the better software ecosystem (CUDA support is universal across all inference engines), but AMD often offers more VRAM per dollar. If you are on Linux and comfortable with ROCm configuration, AMD cards like the RX 7900 XTX (24 GB) provide excellent value. On Windows or if you want zero-friction setup, NVIDIA is the safer choice. Intel Arc is a distant third but works with the SYCL backend.

Should I buy an RTX 5090 or wait for the next generation?

If you need local LLM capability now, the RTX 5090 is an excellent card. The next generation is likely a year or more away, and waiting means a year without local inference capability. GPU technology improves incrementally: the next gen will likely offer 10-20% more bandwidth and possibly more VRAM, but not a generational leap. Buy what you need when you need it, especially if you can find a used RTX 4090 at a good price.

Can I use my gaming GPU for LLMs?

Absolutely. Any NVIDIA, AMD, or Intel GPU with 8+ GB of VRAM can run local LLMs. The same RTX 4070 you game on handles 7-8B models at Q4_K_M. You do not need a separate card for inference unless you want to game and run inference simultaneously. The GPU's VRAM is the only spec that matters differently for LLMs versus gaming.

Why is VRAM more important than TFLOPS for LLMs?

During LLM inference, the GPU reads the entire model weights from VRAM for every token generated. This makes inference memory-bandwidth bound, not compute-bound. A GPU with twice the TFLOPS but the same bandwidth will not generate tokens meaningfully faster. The VRAM capacity determines the ceiling (which models fit), and bandwidth determines the throughput (tokens per second). TFLOPS matters mainly for the prompt processing phase.

Is it worth buying an older used enterprise GPU like a Tesla V100?

The Tesla V100 (16 or 32 GB) can work for LLM inference, but it uses an older architecture with limited software support in modern inference engines. The A6000 or used A100 are better enterprise options because they support modern CUDA features and have better llama.cpp optimizations. The V100 also draws significant power and generates substantial heat. Unless the price is exceptionally low, newer used cards offer a better experience.

Best GPU for Running LLMs Locally in 2026

CanItRunJune 28, 202611 min readHardware

What Matters for LLM Inference: VRAM, Bandwidth, and Compute

LLM inference is fundamentally different from gaming or even from LLM training, and the GPU specs that matter most are different too. For inference, the priority order is: VRAM capacity first, memory bandwidth second, compute (FLOPS) third. VRAM determines which models fit on your GPU. If a model's weights plus KV cache exceed your VRAM, it either does not run or falls back to painfully slow CPU offloading. No amount of compute speed can compensate for insufficient VRAM. Memory bandwidth determines how fast you generate tokens. During autoregressive generation, the GPU reads the entire model weights from memory for every token produced. A model with 40 GB of weights at Q4_K_M needs the GPU to read 40 GB from memory per token. At 1000 GB/s bandwidth (RTX 4090), that is a theoretical maximum of 25 tokens per second. At 500 GB/s (M4 Max), it is 12.5 tokens per second. The actual numbers are lower due to overhead, but bandwidth is the direct bottleneck. Compute (TFLOPS) is the least important for inference. The matrix multiplications during generation are memory-bound, not compute-bound, because the batch size is 1 (one token at a time). Compute matters more for prompt processing (prefill) where the GPU processes all input tokens in parallel, but prefill is usually a small fraction of total inference time.

NVIDIA Consumer GPUs for LLMs

NVIDIA dominates the local LLM space due to broad software support (CUDA, cuBLAS) and a range of VRAM options. The RTX 5090 with 32 GB is the current flagship consumer card. It handles 7-14B models at high quantization with generous context, and can squeeze in some 27-32B models at Q4_K_M with moderate context. Its 1792 GB/s bandwidth makes it the fastest consumer card for token generation. At roughly $2000 MSRP, it is not cheap, but it is the single best consumer GPU for local LLMs. The RTX 4090 with 24 GB remains excellent, especially on the used market where prices have dropped. Twenty-four gigabytes handles up to about 32B at Q3_K_M or 14B at Q8_0. Its 1008 GB/s bandwidth keeps generation speed high. The RTX 4080 with 16 GB is mid-tier: comfortable for 14B models at Q4_K_M but cannot reach the 27B class. The RTX 4060 Ti 16GB is the budget option with 16 GB of VRAM, though its lower bandwidth (288 GB/s) means noticeably slower generation than the 4080. For VRAM-constrained models, it gives the same results, just slower. The RTX 4060 with 8 GB is the entry point, limited to 7B models. Its low price makes it accessible, but upgrading to 12 or 16 GB pays dividends quickly.

AMD Consumer GPUs: The VRAM Advantage

AMD's RDNA 3 lineup offers one significant advantage: more VRAM per dollar on certain models. The RX 7900 XTX with 24 GB and 960 GB/s bandwidth competes directly with the RTX 4090 on VRAM capacity at a lower price point. The RX 7900 XT has 20 GB, a tier that no NVIDIA consumer card occupies. These extra gigabytes can make the difference between fitting a model and not. The caveat is software. AMD GPUs use the ROCm stack on Linux for GPU acceleration in llama.cpp, and ROCm support has improved significantly but is still not as seamless as CUDA. On Windows, the Vulkan backend is the primary option, and it is slower than native CUDA. If you are comfortable with Linux and willing to work through occasional ROCm configuration issues, AMD cards offer excellent value. If you need plug-and-play ease, NVIDIA's CUDA ecosystem is more mature. Intel's Arc series (A770 with 16 GB) is another option, supported via the SYCL backend in llama.cpp. Performance is behind both NVIDIA and AMD, but the A770's 16 GB at a budget price makes it viable for users who already own one.

Apple Silicon: Unified Memory Changes the Game

Apple Silicon deserves special consideration because its unified memory architecture breaks the VRAM ceiling of discrete GPUs. While the best consumer NVIDIA card tops out at 32 GB of VRAM, the M4 Max MacBook Pro is available with 128 GB of unified memory, and the Mac Studio with M4 Ultra reaches 192 GB. This is the only consumer-class hardware that can run Llama 3.3 70B at Q8_0 or larger models at practical quantization levels on a single device. The speed trade-off is real: Apple Silicon's memory bandwidth is lower than high-end NVIDIA GPUs. The M4 Max delivers about 546 GB/s compared to the RTX 5090's 1792 GB/s. Token generation is correspondingly slower. But for models that do not fit on any single consumer GPU, slow inference on Apple Silicon beats no inference at all. The value proposition depends on your model target. If you primarily run 7-14B models, a PC with an RTX 4090 or 5090 is faster and cheaper. If you need 70B models locally, Apple Silicon with 48+ GB is the most practical consumer path. The Mac Mini with M4 Pro and 48 GB is particularly compelling as a dedicated inference machine: it is compact, quiet, energy-efficient, and powerful enough for 70B at Q4_K_M.

Used Enterprise GPUs: A6000, A100, and Others

The used enterprise GPU market is worth exploring for local LLM enthusiasts. The NVIDIA A6000 with 48 GB of VRAM regularly appears on the used market at prices significantly below its original MSRP. Forty-eight gigabytes handles Llama 3.3 70B at Q4_K_M comfortably, which no consumer NVIDIA card can match. The A6000's bandwidth (768 GB/s) is lower than the RTX 4090, so generation speed is moderate, but the sheer VRAM capacity opens up model sizes that consumer cards cannot touch. The A100 40GB and A100 80GB also appear on the used market as cloud providers refresh their fleets. An A100 80GB is overkill for most individual users, but at the right price, it runs any model up to 70B at Q8_0 with room to spare. Considerations for enterprise cards: they typically require more power (300W+), may not have display outputs (irrelevant for headless inference servers), and need PCIe 4.0 x16 slots. Some need active cooling modifications for desktop cases. Also verify the provenance of used datacenter cards. Check warranty status and buy from reputable sellers. Despite the caveats, a used A6000 is an excellent option in the local LLM hardware ecosystem for users who need more VRAM than consumer cards offer but do not want Apple Silicon.

Multi-GPU Considerations

Running multiple GPUs for LLM inference is possible but comes with complications. The most common approach is layer splitting: the model's layers are distributed across GPUs, with each GPU processing its subset. llama.cpp and its derivatives support this natively for NVIDIA GPUs. The practical requirements are a motherboard with multiple PCIe x16 slots (or at least x8 electrical), a power supply that can handle the combined draw (two RTX 4090s need 900W+ for the GPUs alone), and physical clearance for two or three large cards. Performance scales sublinearly: two GPUs do not double your speed because of inter-GPU communication latency. Each time a layer boundary crosses GPU boundaries, data must transfer over PCIe, which is orders of magnitude slower than GPU memory. For two GPUs, expect roughly 1.5x the speed of one. The primary benefit of multi-GPU is not speed but VRAM pooling: two RTX 3090s give you 48 GB of effective VRAM, enough for 70B at Q4_K_M. For most users, a single higher-VRAM card (or Apple Silicon) is simpler and often more cost-effective than a multi-GPU setup. Multi-GPU makes the most sense when you already own one card and can add a second cheaply.

Recommendations by Budget

Under $200: Look at the used market for RTX 3060 12GB cards, which frequently appear at this price point. Twelve gigabytes handles 7B at Q8_0 and tight-fit 14B at Q3_K_M. The value per dollar is hard to beat. $200-400: The RTX 4060 (8GB) or used RTX 3080 (10GB) occupy this tier. If you can find an AMD card with more VRAM at this price, prioritize VRAM over speed at this budget. $400-800: The RTX 4060 Ti 16GB, used RTX 4080 16GB, or RX 7900 XT 20GB are the options. All three handle 14B models comfortably. The RX 7900 XT offers the most VRAM for the money if you are on Linux. $800-1500: The used RTX 4090 (24GB) is the sweet spot here. Its combination of 24 GB VRAM and 1008 GB/s bandwidth makes it the most balanced card in this range. Used A6000s sometimes appear at the top of this range as well. $1500-2500: A new RTX 5090 (32GB), a used A6000 (48GB), or a Mac Mini M4 Pro (48GB unified). The choice depends on whether you prioritize speed (5090), VRAM (A6000), or convenience (Mac Mini). Above $2500: Mac Studio with M4 Max (128GB) or M4 Ultra (192GB), or a dual-GPU NVIDIA setup. This is the territory where you can run any open-weight model locally.

Frequently asked questions

Is NVIDIA or AMD better for running LLMs locally?: NVIDIA has the better software ecosystem (CUDA support is universal across all inference engines), but AMD often offers more VRAM per dollar. If you are on Linux and comfortable with ROCm configuration, AMD cards like the RX 7900 XTX (24 GB) provide excellent value. On Windows or if you want zero-friction setup, NVIDIA is the safer choice. Intel Arc is a distant third but works with the SYCL backend.
Should I buy an RTX 5090 or wait for the next generation?: If you need local LLM capability now, the RTX 5090 is an excellent card. The next generation is likely a year or more away, and waiting means a year without local inference capability. GPU technology improves incrementally: the next gen will likely offer 10-20% more bandwidth and possibly more VRAM, but not a generational leap. Buy what you need when you need it, especially if you can find a used RTX 4090 at a good price.
Can I use my gaming GPU for LLMs?: Absolutely. Any NVIDIA, AMD, or Intel GPU with 8+ GB of VRAM can run local LLMs. The same RTX 4070 you game on handles 7-8B models at Q4_K_M. You do not need a separate card for inference unless you want to game and run inference simultaneously. The GPU's VRAM is the only spec that matters differently for LLMs versus gaming.
Why is VRAM more important than TFLOPS for LLMs?: During LLM inference, the GPU reads the entire model weights from VRAM for every token generated. This makes inference memory-bandwidth bound, not compute-bound. A GPU with twice the TFLOPS but the same bandwidth will not generate tokens meaningfully faster. The VRAM capacity determines the ceiling (which models fit), and bandwidth determines the throughput (tokens per second). TFLOPS matters mainly for the prompt processing phase.
Is it worth buying an older used enterprise GPU like a Tesla V100?: The Tesla V100 (16 or 32 GB) can work for LLM inference, but it uses an older architecture with limited software support in modern inference engines. The A6000 or used A100 are better enterprise options because they support modern CUDA features and have better llama.cpp optimizations. The V100 also draws significant power and generates substantial heat. Unless the price is exceptionally low, newer used cards offer a better experience.