How much VRAM does the NVIDIA RTX 4090 have?

The NVIDIA RTX 4090 has 24 GB of GDDR6X with 1008 GB/s memory bandwidth.

What is the NVIDIA RTX 4090 best for?

With 24 GB of VRAM, the NVIDIA RTX 4090 is well-suited for running 7B–32B models at Q4 with room for context, making it a great all-rounder for local LLM inference.

What LLMs can the NVIDIA RTX 4090 run locally?

The NVIDIA RTX 4090 can run 52 of the 97 open-weight models tracked by CanItRun natively in VRAM at 8k context. Top options include: Qwen 3.8 27B at Q5_K_M, Ornith 1.5 35B-A3B (MoE) at Q3_K_M, Ornith 1.5 9B at BF16.

Can the NVIDIA RTX 4090 run Gemma 4 31B?

Yes. The NVIDIA RTX 4090 runs Gemma 4 31B natively in VRAM at Q4_K_M quantization, achieving approximately 32.4 tokens per second.

Can the NVIDIA RTX 4090 run Qwen 3.6 27B?

Yes. The NVIDIA RTX 4090 runs Qwen 3.6 27B natively in VRAM at Q5_K_M quantization, achieving approximately 33.2 tokens per second.

Can the NVIDIA RTX 4090 run Qwen3 8B?

Yes. The NVIDIA RTX 4090 runs Qwen3 8B natively in VRAM at BF16 quantization, achieving approximately 38.1 tokens per second.

Can I rent the NVIDIA RTX 4090 instead of buying it?

Yes: RunPod and similar cloud GPU providers let you rent NVIDIA RTX 4090 instances by the hour, with no long-term contract. This is often cheaper than buying if you only need it occasionally, and lets you try the GPU before committing to a purchase.

NVIDIA RTX 4090

The NVIDIA RTX 4090 has 24 GB VRAM and 1008 GB/s memory bandwidth. It can run 52 of our 97 tracked models natively in VRAM at 8k context.

With 24 GB GDDR6X, the NVIDIA RTX 4090 is a consumer-tier GPU that can run 52 models natively. This site's calculator puts Llama 3.1 8B at 110.2 tok/s at Q4_K_M (6.66 GB, fits with room to spare) and Qwen3 14B at its recommended Q8_0 quant at 38.4 tok/s (19.12 GB), both comfortably inside the 24GB budget with headroom for context. The real ceiling sits around 32-34B: Qwen3 32B and Yi 1.5 34B both need Q3_K_M to fit natively (19.17 GB at 38.3 tok/s, and 20.79 GB at 35.3 tok/s) at 8k context; stepping up to the more common Q4_K_M tips both into CPU offload, but only barely (23.88 GB and 25.72 GB against this card's ~22.8 GB usable budget), so the falloff is modest rather than severe: Qwen3 32B still holds 30.7 tok/s offloaded, well short of a smaller card's usual cliff. MoE models push further: Qwen3 30B-A3B fits natively at Q4_K_M (21.36 GB, 95.0 tok/s) and Qwen3.5 35B-A3B fits at Q3_K_M (19.04 GB, 131.6 tok/s), despite having more total parameters than the dense 32-34B models above, since only about 3B of those parameters activate per token. GPT-OSS 20B, a model OpenAI explicitly sized for a 16GB card, fits here with room to spare at Q6_K (19.54 GB, 65.2 tok/s). 70B-class dense models (Llama 3.3 70B, Qwen2.5 72B, Llama 3.1 70B) don't fit at any quant at 8k context: Llama 3.3 70B's smallest real build (Q2_K, 32.88 GB) still needs partial offload, decoding at just 3.4 tok/s. GPT-OSS 120B doesn't fit at all, even offloaded: its 62.6 GB MXFP4 weight floor alone exceeds this card's entire ~48.8 GB usable VRAM+RAM budget.

The NVIDIA RTX 4090 launched October 12, 2022 at a $1,599 MSRP, the flagship Ada Lovelace consumer GPU, and for three years the card against which every high-end local LLM setup was measured. It pairs 24GB of GDDR6X on a 384-bit bus (1,008 GB/s bandwidth) with a cut-down AD102 die: 16,384 CUDA cores and 512 4th-gen Tensor Cores from the same 76.3-billion-transistor, 608mm² silicon that tops out at 18,432 cores and 568 Tensor Cores in its uncut configuration. The 4th-gen Tensor Cores added hardware FP8 precision support, a first for a consumer GPU, though using it takes a framework like TensorRT-LLM or vLLM rather than the GGUF quantization most local setups run.

NVIDIA RTX 4090: This is the one GPU on this site whose own real-world benchmarks are baked into the math: this site's core decode-efficiency constant (the number every GPU's tokens/sec estimate is built from) was calibrated directly against a published RTX 4090 llama.cpp benchmark run (hardware-corner.net), not assumed from a spec sheet, and this site's own curve tracks that same source closely on this exact card (see the throughput chart below). The other headline: years after Ada Lovelace gave way to Blackwell, this card is still the highest-VRAM, highest-bandwidth desktop GPU on the market apart from NVIDIA's own $1,999 RTX 5090 flagship: its 24GB and 1,008 GB/s beat every other current Blackwell desktop card, including the newer $999 RTX 5080 (see the VRAM-position chart below).

This site's calculator puts Llama 3.1 8B at 110.2 tok/s at Q4_K_M (6.66 GB, fits with room to spare) and Qwen3 14B at its recommended Q8_0 quant at 38.4 tok/s (19.12 GB), both comfortably inside the 24GB budget with headroom for context. The real ceiling sits around 32-34B: Qwen3 32B and Yi 1.5 34B both need Q3_K_M to fit natively (19.17 GB at 38.3 tok/s, and 20.79 GB at 35.3 tok/s) at 8k context; stepping up to the more common Q4_K_M tips both into CPU offload, but only barely (23.88 GB and 25.72 GB against this card's ~22.8 GB usable budget), so the falloff is modest rather than severe: Qwen3 32B still holds 30.7 tok/s offloaded, well short of a smaller card's usual cliff. MoE models push further: Qwen3 30B-A3B fits natively at Q4_K_M (21.36 GB, 95.0 tok/s) and Qwen3.5 35B-A3B fits at Q3_K_M (19.04 GB, 131.6 tok/s), despite having more total parameters than the dense 32-34B models above, since only about 3B of those parameters activate per token. GPT-OSS 20B, a model OpenAI explicitly sized for a 16GB card, fits here with room to spare at Q6_K (19.54 GB, 65.2 tok/s). 70B-class dense models (Llama 3.3 70B, Qwen2.5 72B, Llama 3.1 70B) don't fit at any quant at 8k context: Llama 3.3 70B's smallest real build (Q2_K, 32.88 GB) still needs partial offload, decoding at just 3.4 tok/s. GPT-OSS 120B doesn't fit at all, even offloaded: its 62.6 GB MXFP4 weight floor alone exceeds this card's entire ~48.8 GB usable VRAM+RAM budget.

Full CUDA support across every major local-inference tool: llama.cpp, Ollama, LM Studio, vLLM, and TensorRT-LLM all treat Ada Lovelace as a mature, first-class target, unlike Blackwell's rockier early sm_120 rollout. GGUF remains the most widely distributed quantization format; ExLlamaV2's EXL2, which used to compete with it, is now archived ("development continues on ExLlamaV3," per the project's own README); EXL3 is the actively maintained successor, targeting NVIDIA Tensor Cores directly down to 1-8 bits per weight. This card's 4th-gen Tensor Cores add hardware FP8 support, which TensorRT-LLM and vLLM both accelerate natively on Ada and Hopper, but not NVFP4: that 4-bit format is Blackwell-only in this site's calculator, gated on GPU architecture rather than just CUDA support, so Q2_K stays this card's smallest available quant on the standard ladder. Practically, that means llama.cpp/Ollama with GGUF is still the default, best-tested path on this card; EXL3 and FP8-native runtimes are the higher-effort options for squeezing out extra throughput once GGUF's ceiling isn't enough.

Vendor	NVIDIA
Architecture	Ada Lovelace
VRAM	24 GB
Memory type	GDDR6X
Memory bandwidth	1008 GB/s
Compute backend	CUDA
Tier	Consumer
Released	2022
Models (native)	52 / 97
Models (offload)	7 / 97

Software: Full llama.cpp and Ollama support out of the box. CUDA 12.x recommended; driver ≥ 525 required.

Three years old, and still ahead of every Blackwell desktop card but one

Plotting this card against the full desktop Blackwell stack this site tracks shows something unusual for a three-generations-old part: it isn't at the bottom of the chart.

VRAM and memory bandwidth, from each card's real spec sheet. A card further right holds bigger models; a card further up decodes them faster once they fit.

This card (this page) still out-VRAMs and out-bandwidths six of the seven current Blackwell desktop GPUs shown here; only NVIDIA's own $1,999 flagship RTX 5090 actually beats it on either axis, with 33.3% more VRAM (32GB vs 24GB) and 77.8% more bandwidth (1,792 vs 1,008 GB/s). Every other Blackwell card trails this one on both counts: the RTX 5080 ($999 launch) has 50% less VRAM (16GB) and 5.0% less bandwidth (960 GB/s); the RTX 5070 Ti and RTX 5060 Ti 16GB share that same 50%-smaller 16GB, with the 5070 Ti also 12.5% slower (896 GB/s) and the 5060 Ti 16GB at less than half this card's bandwidth (448 GB/s). The RTX 5070 (12GB, 672 GB/s) and the RTX 5060/RTX 5050 (8GB, 448/320 GB/s) fall further still; this card holds 2-3x their VRAM and up to 3.15x their bandwidth. A full architecture generation later, only NVIDIA's own flagship actually supersedes it.

The quiet generation before it, and the big jump after it

This card's own bandwidth story only makes sense next to its neighbors: a barely-there gain over the RTX 3090 it replaced, then the biggest bandwidth jump across NVIDIA's last three 24GB+ flagships on the very next generation. Tracking VRAM and bandwidth across all three:

NVIDIA RTX 3090

936 GB/s

NVIDIA RTX 4090 (this page)

1008 GB/s

NVIDIA RTX 5090

1792 GB/s

This card's step up from the RTX 3090 it replaced was surprisingly quiet: bandwidth grew just 7.7% (936 to 1,008 GB/s) at an unchanged 24GB VRAM, nearly a placeholder generation on the number that matters most for decode speed, even though the die, cores, and process node all changed underneath. What came next flipped that hard: the RTX 5090 jumped 77.8% past this card (1,008 to 1,792 GB/s) plus 33.3% more VRAM (24GB to 32GB), the largest single-generation bandwidth gain across NVIDIA's last three 24GB+ flagships. This site's calculator tracks that chain directly on Llama 3.1 8B at Q4_K_M: 102.3 tok/s on the RTX 3090, 110.2 tok/s on this card (a 7.7% gain, matching the bandwidth increase almost exactly), and 195.9 tok/s on the RTX 5090 (a 77.8% gain, again tracking bandwidth almost exactly). One thing didn't change across any of the three: none of them run a 70B-class model natively at this site's 8k-context benchmark; the RTX 5090's bandwidth jump made what already fit decode faster, it didn't expand what fits.

How fast this card actually decodes as context fills up

This site's core decode-efficiency constant was itself calibrated against a real, published RTX 4090 benchmark run, which makes this the one GPU page where comparing this site's own curve to that same independent source is the most direct check available. Plotting Llama 3.1 8B at Q4_K_M's decode speed against context length, from this site's calculator and from hardware-corner.net's own published RTX 4090 run:

This site's calculator (NVIDIA RTX 4090)hardware-corner.net's published RTX 4090 benchmark

Real-world curve: hardware-corner.net's published RTX 4090 LLM benchmark suite for Llama 3.1 8B at Q4_K_XL (llama.cpp build 1bb4f433, flash attention enabled, prefill-then-generate-128-tokens methodology, tested on a system with 129GB of RAM). This site's own decode-efficiency constant was originally calibrated against an earlier run in the same benchmark family.

Both curves fall by almost exactly the same amount over the full range: this site's estimate drops 75.5% (121.1 to 29.7 tok/s) from 4K to 131K context, and hardware-corner.net's real run drops 75.4% (131.0 to 32.3 tok/s) over the same span. This site's numbers run a consistent 2.9-8.2% below the published figures at every point, which tracks with modeling the more common Q4_K_M quant rather than hardware-corner.net's slightly larger Q4_K_XL build, not a real accuracy gap. The one place they genuinely diverge: this site's model predicts weights and KV cache overtake this card's ~22.8 GB usable VRAM budget at around 118,000 tokens of context, tipping from a "fits" verdict into CPU offload; the published real-world run shows no equivalent cliff at its own 131K data point, likely because that benchmark had 129GB of system RAM available as a backstop rather than running purely in VRAM at that length.

Cloud GPU Rental

Don't want to buy a NVIDIA RTX 4090? RunPod is a cloud GPU rental service: rent one by the hour instead, no contract, no upfront hardware cost.

Pay by the hour · no contract · pods start in about a minute.

Rent a NVIDIA RTX 4090 on RunPod ↗ (+$5 signup credit)

Affiliate link: CanItRun may earn a commission. Doesn't affect the fit calculation above.

Popular models for this GPU

Mixtral 8x7B Instruct v0.1 Qwen 3.5 35B-A3B (MoE)Qwen 3.6 35B Ornith 1.5 35B-A3B (MoE)Yi 1.5 34B Chat

Models this GPU runs natively in VRAM (52)

Show 47 more

Models that fit with CPU offload (7)

These use system RAM for layers that don't fit in VRAM, so expect much slower inference.

Too large for this GPU (38)

Compare NVIDIA RTX 4090 with other GPUs

Continue reading

vram-guides9 min

How Much VRAM Does Llama 3 Need? Complete Guide

tutorials10 min

GGUF Quantization Explained: Q4, Q5, Q6, Q8 Compared

hardware11 min

Best GPU for Running LLMs Locally in 2026

tutorials9 min

Getting Started with Ollama: Run Any LLM in One Command

comparisons9 min

GGUF vs EXL2 vs AWQ: Which Quantization Format to Use

vram-guides9 min

Best LLMs for 24 GB VRAM (2026)

vram-guides10 min

Llama Family Guide: Llama 3 to Llama 4 (VRAM & Hardware)

vram-guides10 min

Qwen Family Guide: 2.5 to 3.6

vram-guides10 min

DeepSeek Family Guide: R1, V3 & Distilled Models

vram-guides9 min

Mistral & Mixtral Family: Complete VRAM Reference

hardware10 min

Best GPUs for Coding Agents in 2026

hardware11 min

Multi-GPU Setups for Local LLMs: The Complete Guide

hardware10 min

How to Choose Your First GPU for Local LLMs

tutorials9 min

Getting Started with LM Studio: GUI-Based Local LLMs

tutorials12 min

llama.cpp Complete Setup Guide: Build, Configure, and Optimize

tutorials9 min

How to Benchmark Your Local LLM Setup

comparisons10 min

Ollama vs LM Studio vs llama.cpp: Which Tool Should You Use?

hardware10 min

Best GPUs $1000+ for LLMs (2026)

hardware10 min

Used Enterprise GPUs for LLMs: A6000, A100, and Beyond

hardware10 min

NVIDIA RTX 40 Series for LLMs: Complete VRAM & Performance Guide

tutorials11 min

Getting Started with Local LLMs (2026): The Complete Beginner's Guide

tutorials10 min

LLM Benchmarks Explained: MMLU, HumanEval, SWE-bench, and More

Frequently asked questions

How much VRAM does the NVIDIA RTX 4090 have?: The NVIDIA RTX 4090 has 24 GB of GDDR6X with 1008 GB/s memory bandwidth.
What is the NVIDIA RTX 4090 best for?: With 24 GB of VRAM, the NVIDIA RTX 4090 is well-suited for running 7B–32B models at Q4 with room for context, making it a great all-rounder for local LLM inference.
What LLMs can the NVIDIA RTX 4090 run locally?: The NVIDIA RTX 4090 can run 52 of the 97 open-weight models tracked by CanItRun natively in VRAM at 8k context. Top options include: Qwen 3.8 27B at Q5_K_M, Ornith 1.5 35B-A3B (MoE) at Q3_K_M, Ornith 1.5 9B at BF16.
Can the NVIDIA RTX 4090 run Gemma 4 31B?: Yes. The NVIDIA RTX 4090 runs Gemma 4 31B natively in VRAM at Q4_K_M quantization, achieving approximately 32.4 tokens per second.
Can the NVIDIA RTX 4090 run Qwen 3.6 27B?: Yes. The NVIDIA RTX 4090 runs Qwen 3.6 27B natively in VRAM at Q5_K_M quantization, achieving approximately 33.2 tokens per second.
Can the NVIDIA RTX 4090 run Qwen3 8B?: Yes. The NVIDIA RTX 4090 runs Qwen3 8B natively in VRAM at BF16 quantization, achieving approximately 38.1 tokens per second.
Can I rent the NVIDIA RTX 4090 instead of buying it?: Yes: RunPod and similar cloud GPU providers let you rent NVIDIA RTX 4090 instances by the hour, with no long-term contract. This is often cheaper than buying if you only need it occasionally, and lets you try the GPU before committing to a purchase.