How much VRAM does the NVIDIA A100 40GB have?

The NVIDIA A100 40GB has 40 GB of HBM2 with 1555 GB/s memory bandwidth.

What is the NVIDIA A100 40GB best for?

With 40 GB of VRAM, the NVIDIA A100 40GB is well-suited for running 7B–32B models at Q4 with room for context, making it a great all-rounder for local LLM inference.

What LLMs can the NVIDIA A100 40GB run locally?

The NVIDIA A100 40GB can run 47 of the 70 open-weight models tracked by CanItRun natively in VRAM at 8k context. Top options include: Llama 3.3 70B Instruct at Q3_K_M, Llama 3.1 8B Instruct at FP32, Llama 3.2 3B Instruct at FP32.

Can the NVIDIA A100 40GB run Llama 3.3 70B Instruct?

Yes. The NVIDIA A100 40GB runs Llama 3.3 70B Instruct natively in VRAM at Q3_K_M quantization, achieving approximately 51.7 tokens per second.

Can the NVIDIA A100 40GB run Qwen 3.6 27B?

Yes. The NVIDIA A100 40GB runs Qwen 3.6 27B natively in VRAM at NVFP4 quantization, achieving approximately 115.2 tokens per second.

Can the NVIDIA A100 40GB run Llama 3.1 8B Instruct?

Yes. The NVIDIA A100 40GB runs Llama 3.1 8B Instruct natively in VRAM at FP32 quantization, achieving approximately 48.6 tokens per second.

NVIDIA A100 40GB

The NVIDIA A100 40GB has 40 GB VRAM and 1555 GB/s memory bandwidth. It can run 47 of our 70 tracked models natively in VRAM at 8k context.

With 40 GB HBM2, the NVIDIA A100 40GB is a datacenter-tier GPU that can run 47 models natively. It handles 70B-class models at Q4 quantization.

The NVIDIA A100 40GB is the PCIe variant of NVIDIA's Ampere data center GPU, the predecessor to the H100. Its 40GB of HBM2 memory and 1,555 GB/s bandwidth make it well-suited for running 7B–34B models at high throughput in cloud or on-prem inference clusters. This is the GPU behind many AWS p4d and Azure NDv4 instances and remains widely deployed in production LLM serving.

NVIDIA A100 40GB: 40GB HBM2 at 1555 GB/s — PCIe variant for cloud instances.

32B at Q4 native. 70B at Q4 with CPU offload. ~30-50 t/s for 7B.

Full CUDA support. Common cloud GPU for scaled inference deployments.

Vendor	NVIDIA
Architecture	Ampere
VRAM	40 GB
Memory type	HBM2
Memory bandwidth	1555 GB/s
Compute backend	CUDA
Tier	Datacenter
Released	2020
Models (native)	47 / 70
Models (offload)	7 / 70

Software: Typically cloud-accessed. vLLM and TensorRT-LLM give best batched-inference performance.

Popular models for this GPU

Qwen 2.5 72B Instruct Llama 3.3 70B Instruct DeepSeek R1 Distill Llama 70B Llama 3.1 70B Instruct Mixtral 8x7B Instruct v0.1

Models this GPU runs natively in VRAM (47)

Models that fit with CPU offload (7)

These use system RAM for layers that don't fit in VRAM — expect much slower inference.

Too large for this GPU (16)

Frequently asked questions

How much VRAM does the NVIDIA A100 40GB have?: The NVIDIA A100 40GB has 40 GB of HBM2 with 1555 GB/s memory bandwidth.
What is the NVIDIA A100 40GB best for?: With 40 GB of VRAM, the NVIDIA A100 40GB is well-suited for running 7B–32B models at Q4 with room for context, making it a great all-rounder for local LLM inference.
What LLMs can the NVIDIA A100 40GB run locally?: The NVIDIA A100 40GB can run 47 of the 70 open-weight models tracked by CanItRun natively in VRAM at 8k context. Top options include: Llama 3.3 70B Instruct at Q3_K_M, Llama 3.1 8B Instruct at FP32, Llama 3.2 3B Instruct at FP32.
Can the NVIDIA A100 40GB run Llama 3.3 70B Instruct?: Yes. The NVIDIA A100 40GB runs Llama 3.3 70B Instruct natively in VRAM at Q3_K_M quantization, achieving approximately 51.7 tokens per second.
Can the NVIDIA A100 40GB run Qwen 3.6 27B?: Yes. The NVIDIA A100 40GB runs Qwen 3.6 27B natively in VRAM at NVFP4 quantization, achieving approximately 115.2 tokens per second.
Can the NVIDIA A100 40GB run Llama 3.1 8B Instruct?: Yes. The NVIDIA A100 40GB runs Llama 3.1 8B Instruct natively in VRAM at FP32 quantization, achieving approximately 48.6 tokens per second.