CanItRun Logocanitrun.

NVIDIA RTX 4000 Ada

The NVIDIA RTX 4000 Ada has 20 GB VRAM and 320 GB/s memory bandwidth. It can run 42 of our 71 tracked models natively in VRAM at 8k context.

With 20 GB GDDR6, the NVIDIA RTX 4000 Ada is a workstation-tier GPU that can run 42 models natively. It handles 30B-class models at Q4 quantization.

The NVIDIA RTX 4000 Ada is the entry-level Ada Lovelace workstation GPU, built on a 160-bit memory bus with 20GB ECC GDDR6 at 320 GB/s. With 6,144 CUDA cores and a single-slot form factor, it is the most compact professional GPU in the Ada lineup. Its 20GB VRAM comfortably handles 13B models at Q8_0 and 14B–20B models at Q4_K_M, exceeding what typical consumer 16GB cards can hold, while fitting into thermally constrained workstations and SFF builds.

The NVIDIA RTX 4000 Ada is a professional workstation NVIDIA GPU based on the Ada Lovelace architecture. Released in 2023. It features 20 GB of GDDR6 at 320 GB/s memory bandwidth. Full llama.cpp and Ollama support out of the box. CUDA 12.x recommended; driver ≥ 525 required.

For local LLM inference, this GPU runs 42 of the 71 models we track natively in VRAM at 8K context. The largest model it handles in VRAM is Mixtral 8x7B Instruct v0.1 (82.9 t/s at Q2_K). It handles smaller models up to ~7-14B at reasonable precision, with some 27-32B models fitting at lower quantization. On Qwen 3.6 27B, it achieves approximately 23.7 tokens per second at NVFP4 quantization. An additional 9 models fit with CPU offload — slower but usable.

NVIDIA's CUDA ecosystem provides broad out-of-the-box support across llama.cpp, Ollama, vLLM, and TensorRT-LLM. Among workstation GPUs, it sits above Apple M5 Pro (24GB) and NVIDIA RTX 4060 Ti 16GB in performance, but below Intel Arc Pro B60 24GB.

VendorNVIDIA
ArchitectureAda Lovelace
VRAM20 GB
Memory typeGDDR6
Memory bandwidth320 GB/s
Compute backendCUDA
TierWorkstation
Released2023
Models (native)42 / 71
Models (offload)9 / 71
Software: Full llama.cpp and Ollama support out of the box. CUDA 12.x recommended; driver ≥ 525 required.

Popular models for this GPU

Mixtral 8x7B Instruct v0.1Qwen 3.5 35B-A3B (MoE)Qwen 3.6 35BYi 1.5 34B ChatQwen3 32B

Models this GPU runs natively in VRAM (42)

Models that fit with CPU offload (9)

These use system RAM for layers that don't fit in VRAM — expect much slower inference.

Too large for this GPU (20)

Models mentioned

Mixtral 8x7B Instruct v0.146.7B (12.9B active)
Mistral AIQ4_K_M rec.
Qwen 3.5 35B-A3B (MoE)35B (3B active)
AlibabaQ4_K_M rec.
Qwen 3.6 35B35B
AlibabaQ4_K_M rec.
Yi 1.5 34B Chat34.4B
01.AIQ4_K_M rec.
Qwen3 32B32.8B
AlibabaQ4_K_M rec.
Qwen 2.5 32B Instruct32.5B
AlibabaQ4_K_M rec.
Qwen 2.5 Coder 32B Instruct32.5B
AlibabaQ4_K_M rec.
DeepSeek R1 Distill Qwen 32B32.5B
DeepSeekQ4_K_M rec.
Nemotron 3 Nano 30B32B (3B active)
NVIDIAQ5_K_M rec.
Gemma 4 31B31B
GoogleQ4_K_M rec.
Qwen3 30B-A3B (MoE)30B (3B active)
AlibabaQ4_K_M rec.
Gemma 2 27B Instruct27.2B
GoogleQ4_K_M rec.
Gemma 3 27B Instruct27B
GoogleQ4_K_M rec.
Qwen 3.6 27B27B
AlibabaQ4_K_M rec.
Gemma 4 26B (MoE)26B (3.8B active)
GoogleQ4_K_M rec.
Mistral Small 3.1 24B Instruct24B
Mistral AIQ4_K_M rec.
Mistral Small 22B22.2B
Mistral AIQ4_K_M rec.
GPT-OSS 20B21B (4B active)
OpenAIQ5_K_M rec.
Qwen3 14B14.8B
AlibabaQ5_K_M rec.
Qwen 2.5 14B Instruct14.7B
AlibabaQ5_K_M rec.
Phi-4 14B Instruct14B
MicrosoftQ5_K_M rec.
Mistral Nemo 12B Instruct12.2B
Mistral AIQ5_K_M rec.
Gemma 3 12B Instruct12.2B
GoogleQ5_K_M rec.
Gemma 2 9B Instruct9.2B
GoogleQ5_K_M rec.
Llama 3.1 8B Instruct8B
MetaQ5_K_M rec.
DeepSeek R1 Distill Llama 8B8B
DeepSeekQ5_K_M rec.
Qwen3 8B8B
AlibabaQ5_K_M rec.
Qwen 2.5 7B Instruct7.6B
AlibabaQ6_K rec.
Mistral 7B Instruct v0.37.25B
Mistral AIQ6_K rec.
Gemma 3 4B Instruct4B
GoogleQ6_K rec.
Gemma 4 E4B4B
GoogleQ5_K_M rec.
Phi-3.5 Mini Instruct3.8B
MicrosoftQ6_K rec.
Llama 3.2 3B Instruct3.2B
MetaQ6_K rec.
Qwen 2.5 3B Instruct3.1B
AlibabaQ6_K rec.
Gemma 2 2B Instruct2.6B
GoogleQ8_0 rec.
Gemma 4 E2B2B
GoogleQ8_0 rec.
SmolLM2 1.7B Instruct1.7B
Hugging FaceQ8_0 rec.
Qwen 2.5 1.5B Instruct1.5B
AlibabaQ8_0 rec.
Llama 3.2 1B Instruct1.24B
MetaQ8_0 rec.
Gemma 3 1B Instruct1B
GoogleQ8_0 rec.
Qwen 2.5 0.5B Instruct0.5B
AlibabaQ8_0 rec.
SmolLM2 360M Instruct0.36B
Hugging FaceQ8_0 rec.

Frequently asked questions

How much VRAM does the NVIDIA RTX 4000 Ada have?
The NVIDIA RTX 4000 Ada has 20 GB of GDDR6 with 320 GB/s memory bandwidth.
What is the NVIDIA RTX 4000 Ada best for?
With 20 GB of VRAM, the NVIDIA RTX 4000 Ada handles smaller models (7B–14B) at Q4–Q5 quantization — ideal for entry-level local LLM experimentation and lightweight inference.
What LLMs can the NVIDIA RTX 4000 Ada run locally?
The NVIDIA RTX 4000 Ada can run 42 of the 71 open-weight models tracked by CanItRun natively in VRAM at 8k context. Top options include: Llama 3.1 8B Instruct at NVFP4, Llama 3.2 3B Instruct at FP32, Llama 3.2 1B Instruct at FP32.
Can the NVIDIA RTX 4000 Ada run Llama 3.3 70B Instruct?
The NVIDIA RTX 4000 Ada can run Llama 3.3 70B Instruct with CPU offload at NVFP4 quantization, but inference will be slower than native VRAM execution.
Can the NVIDIA RTX 4000 Ada run Qwen 3.6 27B?
Yes. The NVIDIA RTX 4000 Ada runs Qwen 3.6 27B natively in VRAM at NVFP4 quantization, achieving approximately 23.7 tokens per second.
Can the NVIDIA RTX 4000 Ada run Llama 3.1 8B Instruct?
Yes. The NVIDIA RTX 4000 Ada runs Llama 3.1 8B Instruct natively in VRAM at NVFP4 quantization, achieving approximately 80 tokens per second.