Can the RTX 5090 really run a 70B model?

At Q2_K quantization (~31 GB), yes. At Q4_K_M (~49 GB total), no, and even 48 GB (dual 3090s, or an A6000) only gets you to Q3_K_M, not Q4_K_M. The 5090 is the first consumer card to cross the 70B threshold, but you are right at the edge, with limited context headroom and a real quality cost. For genuine Q4_K_M-quality 70B inference, three pooled GPUs or 64+ GB of Apple Silicon is recommended.

Is NVFP4 worth waiting for?

Not as a primary reason to buy or wait. NVFP4 will eventually improve quality for 4-bit models on Blackwell, but GGUF Q4_K_M is already very good. Treat NVFP4 as a bonus feature that will improve your Blackwell card over time, not as a must-have today. By the time the NVFP4 ecosystem matures (late 2026-2027), you will already have a compatible GPU if you buy Blackwell now.

RTX 5080 16GB or used RTX 4090 24GB?

These are not the same VRAM tier: the 5080 has 16 GB at 960 GB/s, the 4090 has 24 GB at 1008 GB/s. The 4090's extra 8 GB is the bigger factor for LLM inference, it is the difference between a 16 GB card's 20-27B aggressive-quantization ceiling and native room for 27-32B models at Q4. Buy the 5080 only if you specifically need new-with-warranty and your models already fit in 16 GB. Otherwise, a used 4090 is the better LLM card, even though the 5080 does offer GDDR7 and NVFP4 support.

Will there be an RTX 5090 Ti with more VRAM?

Rumored but unconfirmed as of mid-2026. A 5090 Ti could theoretically use higher-density GDDR7 modules (3 GB per chip instead of 2 GB) to achieve 48 GB on a 512-bit bus. That would still only reach Q3_K_M on a 70B, the same ceiling every real 48 GB card hits; a single consumer card reaching Q4_K_M on a 70B would need meaningfully more than 48 GB. Estimated price: $2500+. No official announcement from NVIDIA.

Is the RTX 5060 good enough for LLMs?

The RTX 5060 8GB and 12GB variants target entry-level gaming. The 8 GB model is limited to 7B at Q4, the same capability as a used $200 RTX 3060 12GB. The 5060 12GB model (if available) would be more interesting. For LLMs, budget Blackwell is less compelling than used Ampere (RTX 3060 12GB, RTX 3080 12GB) at the same or lower prices.

NVIDIA RTX 50 Series (Blackwell) for LLMs: Complete Guide

CanItRunJuly 1, 202610 min readHardware

Blackwell Architecture: What Changed for LLMs

NVIDIA's RTX 50 series (Blackwell, 2025-2026) brings three meaningful improvements for LLM inference. First, GDDR7 memory delivers significantly higher bandwidth: the RTX 5090's 1792 GB/s is a 78% increase over the 4090's 1008 GB/s. This directly translates to 1.5-2x faster token generation for same-size models. Second, the consumer VRAM ceiling rises to 32 GB (RTX 5090), up from 24 GB (RTX 4090). While still well short of the ~49 GB total needed for 70B at Q4_K_M, 32 GB enables 70B at Q2_K (~31 GB), the first time a single consumer card can technically run a 70B model. Third, NVFP4 (native 4-bit floating point) support via 5th-gen Tensor Cores. Unlike GGUF's integer quantization, NVFP4 uses a true 4-bit floating point representation that can preserve more model quality at the same bit-width. The caveat: NVFP4 requires software support that is still maturing. llama.cpp's NVFP4 backend is under development, and Ollama does not yet support it. For now, GGUF Q4_K_M remains the practical standard even on Blackwell. As the software ecosystem catches up, NVFP4 could become a meaningful differentiator. For production inference today, treat NVFP4 as a future-looking feature, not a current reason to buy.

RTX 5090 (32 GB): The New Flagship

The RTX 5090 (32 GB, 1792 GB/s, $2000 MSRP) is the most capable consumer LLM GPU ever made. Thirty-two gigabytes changes the game: for the first time, a consumer card can load a 70B model (Llama 3.3 70B at Q2_K, ~31 GB) with short context, right at the card's effective capacity. At Q4_K_M (~49 GB total), 70B still does not fit, and neither does a single real 48 GB card (an A6000 tops out at Q3_K_M); you need a third pooled GPU or 64+ GB of Apple unified memory for genuine Q4_K_M quality. But Q2_K 70B on a single card is a milestone. For 27-32B models at Q4 (~17-20 GB), the 5090 delivers 55-70+ tok/s, roughly 1.8x faster than the 4090. This speed is transformative for agentic coding workflows where sessions can generate 20K+ tokens: a task that takes 10 minutes on the 4090 takes 5-6 minutes on the 5090. The 512-bit memory bus with 28 Gbps GDDR7 modules achieves the 1792 GB/s figure. NVFP4 tensor core support may eventually improve quality for 4-bit models, but GGUF remains the practical standard. For users building the best possible single-card local LLM rig, the RTX 5090 is the answer. The main argument against it: at $2000, you could instead buy dual RTX 3090s (48 GB, ~$1500-1800) and run 70B at Q3_K_M, a configuration that needs more power and complexity, and still doesn't reach Q4_K_M without a third card.

# RTX 5090: first consumer card to run 70B
ollama run llama3.3:70b-q2_K        # ~31 GB, fits at the card's edge

# 32B models at unprecedented speed
ollama run qwen3.6:27b              # ~17 GB, 55-70 tok/s
ollama run deepseek-r1:32b-q4_K_M   # ~20 GB, 45-55 tok/s

# NVFP4 support (when software matures)
# Will offer better quality than Q4_K_M at similar VRAM

RTX 5080 (16 GB), 5070 Ti (16 GB), 5070 (12 GB)

The RTX 5080 (16 GB, 960 GB/s, $1000-1200 expected) stays at the same 16 GB capacity as the RTX 4080 and RTX 4070 Ti Super, not a jump to 24 GB. For LLM inference that means the same 14B-comfortable, 20-27B-aggressive-quant ceiling as those cards, just faster: 960 GB/s is a 34% bandwidth gain over the RTX 4080's 717 GB/s, and it edges past even the used RTX 3090's 936 GB/s (up 2.6%), though it still falls about 5% short of the RTX 4090's 1008 GB/s. If pricing holds at $1000-1200, that makes it a reasonable new-with-warranty pick within a 16 GB ceiling, not a 24 GB one. The RTX 5070 Ti (16 GB, ~896 GB/s expected, $750-800) continues the pattern of 70-class cards at 16 GB. Excellent for 14B models, good bandwidth, but 16 GB limits model capacity. The RTX 5070 (12 GB, 672 GB/s, $549) is a fast 12 GB card: great bandwidth in its class, but 12 GB caps you at 14B models. For LLMs, 12 GB at $549 is poor value compared to the used RTX 3060 12GB ($200) or Arc B580 12GB ($249). The 5070 is a gaming card. The RTX 5060 Ti 16GB (~$500 expected, specs TBD) could be interesting if it pairs 16 GB with GDDR7 bandwidth significantly above the 4060 Ti's 288 GB/s. Early specifications suggest a 128-bit bus, which would cap bandwidth frustratingly low again.

NVFP4: The Blackwell Wildcard

NVFP4 is NVIDIA's hardware-accelerated 4-bit floating-point format, exclusive to Blackwell GPUs. Unlike GGUF's integer quantization (which maps continuous weight values to discrete integer buckets), NVFP4 uses a true miniaturized floating-point representation: 1 sign bit, 2 exponent bits, 1 mantissa bit. This floating-point approach can theoretically preserve more nuanced weight information than integer quantization at the same bit-width. The Blackwell tensor cores execute NVFP4 operations natively, meaning inference using NVFP4-formatted models can be both memory-efficient (4 bits per weight, same as Q4_K_M) and fast (tensor cores handle the computation rather than CUDA cores). The catch in mid-2026: software support on the Blackwell/CUDA path this guide covers is nascent. llama.cpp gained NVFP4 support through a run of pull requests between late March and late April 2026, but Ollama's native CUDA integration for NVFP4 on desktop Blackwell cards is still being developed. Oddly, Ollama's NVFP4 support actually shipped first on Apple Silicon: its MLX engine has decoded NVFP4 models on M-series Macs, in software rather than via tensor cores, since the March 2026 MLX preview. ExLlamaV2 has preliminary NVFP4 support for some architectures. NVFP4-formatted models on Hugging Face are rare, the ecosystem has not yet produced a broad library of pre-quantized NVFP4 models. For early adopters, NVFP4 is a reason to be excited about Blackwell's future. For production inference today, it is not a factor in purchase decisions. By late 2026 or early 2027, NVFP4 could become a standard option alongside GGUF Q4_K_M.

Which Blackwell GPU to Buy for LLMs

RTX 5090 ($2000): buy if you want the single best consumer GPU for LLMs and have the budget. The 32 GB + 1792 GB/s combination is unmatched. You will run 27-32B models at incredible speed and can touch 70B at Q3. RTX 5080 ($1000-1200): buy if you want the fastest 16 GB NVIDIA card with warranty. It shares its 16 GB ceiling with the RTX 4080 and RTX 4070 Ti Super, but at 960 GB/s it is faster than both; it is not a substitute for the 24 GB the RTX 5090 or a used RTX 3090/4090 provide. RTX 5070 Ti 16GB ($750-800): buy if you need a new 16 GB NVIDIA card for 14B models with strong bandwidth. It is the best 14B-class card in the Blackwell lineup. RTX 5070 12GB ($549): skip for LLMs. Used alternatives (3090, 3060 12GB) are better. RTX 5060 Ti 16GB (~$500): wait for independent bandwidth benchmarks. If bandwidth is still 128-bit (~288 GB/s), skip, the RTX 3060 12GB is faster. If NVIDIA gives it a wider bus, it could be compelling. RTX 5050 and below: skip. Insufficient VRAM for practical LLM inference.

Blackwell (50 Series) vs Ada (40 Series): Upgrade Worth It?

From RTX 4090 to 5090: the upgrade is significant. 24→32 GB opens up 70B at Q3, and 1008→1792 GB/s roughly doubles generation speed. It is worth the upgrade if you run large models daily. From RTX 4080/4070 Ti Super to 5080: 16→16 GB is not a VRAM upgrade at all, the model-size ceiling stays exactly where it was. The bandwidth improvement (717→960 GB/s, up 34%, or 672→960 GB/s, up 43%) is real and speeds up every model both cards already fit, but it will not let you run anything larger. Worth it only if raw speed within the existing 16 GB ceiling matters more to you than VRAM headroom. From RTX 4070/4060 Ti to 5070/5060 Ti: the generation-to-generation VRAM capacities are unchanged (12→12, 16→16). The bandwidth improvements from GDDR7 help speed but do not unlock new model sizes. Not worth upgrading for LLMs alone. From RTX 3090/3080 (Ampere) to 5080/5070 Ti: if you have a 3090 (24 GB), both the 5080 and 5070 Ti (16 GB each) are a VRAM downgrade, not a side-grade. The 5080's bandwidth only edges past the 3090's (936→960 GB/s, up 2.6%), nowhere near enough to offset 8 GB less capacity. From anything older (20 series, 10 series, GTX): any Blackwell above the 5060 is a massive upgrade. The generational leaps in VRAM, bandwidth, and CUDA features are transformative.

Frequently asked questions

Can the RTX 5090 really run a 70B model?: At Q2_K quantization (~31 GB), yes. At Q4_K_M (~49 GB total), no, and even 48 GB (dual 3090s, or an A6000) only gets you to Q3_K_M, not Q4_K_M. The 5090 is the first consumer card to cross the 70B threshold, but you are right at the edge, with limited context headroom and a real quality cost. For genuine Q4_K_M-quality 70B inference, three pooled GPUs or 64+ GB of Apple Silicon is recommended.
Is NVFP4 worth waiting for?: Not as a primary reason to buy or wait. NVFP4 will eventually improve quality for 4-bit models on Blackwell, but GGUF Q4_K_M is already very good. Treat NVFP4 as a bonus feature that will improve your Blackwell card over time, not as a must-have today. By the time the NVFP4 ecosystem matures (late 2026-2027), you will already have a compatible GPU if you buy Blackwell now.
RTX 5080 16GB or used RTX 4090 24GB?: These are not the same VRAM tier: the 5080 has 16 GB at 960 GB/s, the 4090 has 24 GB at 1008 GB/s. The 4090's extra 8 GB is the bigger factor for LLM inference, it is the difference between a 16 GB card's 20-27B aggressive-quantization ceiling and native room for 27-32B models at Q4. Buy the 5080 only if you specifically need new-with-warranty and your models already fit in 16 GB. Otherwise, a used 4090 is the better LLM card, even though the 5080 does offer GDDR7 and NVFP4 support.
Will there be an RTX 5090 Ti with more VRAM?: Rumored but unconfirmed as of mid-2026. A 5090 Ti could theoretically use higher-density GDDR7 modules (3 GB per chip instead of 2 GB) to achieve 48 GB on a 512-bit bus. That would still only reach Q3_K_M on a 70B, the same ceiling every real 48 GB card hits; a single consumer card reaching Q4_K_M on a 70B would need meaningfully more than 48 GB. Estimated price: $2500+. No official announcement from NVIDIA.
Is the RTX 5060 good enough for LLMs?: The RTX 5060 8GB and 12GB variants target entry-level gaming. The 8 GB model is limited to 7B at Q4, the same capability as a used $200 RTX 3060 12GB. The 5060 12GB model (if available) would be more interesting. For LLMs, budget Blackwell is less compelling than used Ampere (RTX 3060 12GB, RTX 3080 12GB) at the same or lower prices.