NVIDIA A100 40GB

NVIDIA A100 40GB: The original A100 configuration, unveiled at GTC in May 2020 as Ampere's debut datacenter GPU, before the 80GB HBM2e variant followed that November. This 40GB card uses the same 826mm² GA100 die (TSMC 7nm, 54.2 billion transistors, 6,912 CUDA cores) as the 80GB card, but with older HBM2 instead of HBM2e, capping bandwidth at 1,555 GB/s, 24% below the 80GB card's 2,039 GB/s despite identical silicon. Ships as either a 400W SXM4 module or a 250W PCIe card, and like the 80GB variant supports Multi-Instance GPU (MIG) partitioning into up to seven isolated slices.

This site's calculator only fits Llama 3.3 70B at the most aggressive Q2_K (32.88 GB, 34.4 tok/s) in 40GB; every higher-quality quant either needs CPU offload or doesn't fit outright. The card is far more comfortable in the 8B-14B range: Qwen 2.5 14B fits natively even at full BF16 (34.73 GB, 32.6 tok/s), and Llama 3.1 8B is small enough to run at full, uncompressed FP32 precision (37.04 GB, 30.6 tok/s), a precision tier most models this site tracks never reach on any single GPU. GPT-OSS 120B (MoE) doesn't fit at any quantization; its smallest build alone needs 62.60 GB of weights. Across the 97 tracked models, 57 fit fully in VRAM at 8k context, 8 fewer than the 80GB card manages with double the capacity.

Full CUDA support under compute capability 8.0 (sm_80), identical software stack to the 80GB card: llama.cpp, vLLM, and TensorRT-LLM all treat the two capacities the same way, just with a lower ceiling. Common on AWS p4d and Azure NDv4 instances, where several 40GB cards get pooled over NVLink (600 GB/s per GPU on SXM4) for models too large for one card. The cheapest way into genuine datacenter-class CUDA hardware on the secondhand market now that 80GB+ cards dominate new deployments, though the smaller ceiling and MIG partitioning make it better suited to serving many small models at once than any single large one.