How to Choose Your First GPU for Local LLMs

The One Rule: VRAM Determines Everything

If you take away one thing from this guide, make it this: VRAM capacity is the single most important specification for running LLMs locally. It determines which models you can run, at what quantization level, and with how much context. Everything else — GPU cores, clock speed, Tensor cores, RT cores — is secondary. A $200 used RTX 3060 with 12 GB of VRAM can run models that a $2000 RTX 4080 with 8 GB cannot, because the model literally does not fit in 8 GB. The math is simple: each billion parameters at Q4_K_M quantization needs roughly 0.6 GB of VRAM for weights alone — add another 0.5-2 GB for KV cache depending on context length. The chart below shows how that scales across common model sizes. When choosing a GPU, always buy the most VRAM you can afford. It is better to have a slower card with more VRAM than a faster card with less VRAM. The slower card runs the same models, just at fewer tokens per second. The faster card with less VRAM cannot run the larger models at all.

Budget Tiers: What Each Price Range Gets You

Here is what your budget realistically gets you, from a $200 used card up to dual-GPU or Apple Silicon setups above $2500. As a rule, take more VRAM over a faster GPU at every tier — see the recommendations section below for the single best pick at each budget.

New vs Used: The Smart Money Is on Used GPUs

GPUs are one of the safest used PC components to buy. They have no moving parts (fans are easily replaceable), they either work or they do not, and they have no consumable components. The used market is particularly favorable for LLM users because miners and gamers upgrade for reasons that do not affect LLM performance. A mined-on RTX 3090 that ran at constant temperature for two years may have more wear on its fans but is electrically fine — and LLM inference draws far less power than mining, so it will run cooler than ever. The sweet spot on the used market is the RTX 3090 at $700-900, which offers 24 GB of VRAM at less than half the cost of a new RTX 4090. The RTX 3060 12GB at $180-250 is the budget champion. Both are readily available because they were popular mining cards during the crypto boom and are now flooding the used market. Before you hand over money, run through this checklist:

• Check that warranty stickers are intact — the card was not disassembled
• Verify it outputs video correctly before paying
• Run a quick VRAM stress test (OCCT or similar)
• Check that all fans spin
• Buy from platforms with buyer protection (eBay, r/hardwareswap with confirmed trades)
• Avoid listings described as 'for parts' or 'untested' — these are almost always dead

Common Mistakes First-Time Buyers Make

First-time buyers tend to make the same handful of mistakes. Here are the big ones — plus the power-supply check almost everyone skips.

1. Buying a GPU based on gaming benchmarks. Gaming performance scales with compute (TFLOPS); LLM inference scales with VRAM capacity and memory bandwidth. An RTX 3060 12GB is the better LLM card than a faster RTX 4060 Ti 8GB, because the extra 4 GB of VRAM matters more than higher clock speeds.
2. Buying insufficient VRAM and planning to 'just use CPU offloading.' Partial offloading works, but it is dramatically slower (1-5 tok/s vs 20-50 tok/s) — a fallback, not a strategy.
3. Ignoring memory bandwidth. VRAM determines which models fit; bandwidth determines how fast they generate. An RTX 4060 Ti 16GB (288 GB/s) generates tokens about 2.5x slower than an RTX 3090 (936 GB/s), even on the same 14B model.

NVIDIA vs AMD vs Intel: The Ecosystem Reality

For a first GPU, NVIDIA is the safest choice — the CUDA ecosystem is universal, so you will spend zero time troubleshooting compatibility and all your time actually using models. AMD (RX 7900 XTX, 24 GB, 960 GB/s at $750-850 used) and Intel (Arc B580, 12 GB, 456 GB/s at $249 new) can offer excellent value once you are comfortable configuring the software stack yourself, which makes them better suited as second GPUs once you understand your specific needs. Here is how the three compare for local LLM work:

Should You Buy a Mac Instead?

For some first-time users, a Mac with Apple Silicon is a better choice than a PC with a discrete GPU — it depends on your priorities. The decision comes down to your model ambitions: if you want to run 70B models, Mac is more practical. If 7-32B models cover your needs, a PC with an NVIDIA GPU is faster and more versatile.

Concrete Recommendations for First-Time Buyers

Putting it all together, here is the single best pick at each budget. Whatever you choose, remember: VRAM first, bandwidth second, everything else a distant third.