Multi-GPU Setups for Local LLMs: The Complete Guide

Why Multi-GPU? The VRAM Ceiling Problem

The single biggest limitation of consumer GPUs for LLM inference is VRAM capacity. The best consumer card (RTX 5090) tops out at 32 GB. Meanwhile, Llama 3.3 70B at Q4_K_M needs about 40 GB, Qwen 2.5 72B at Q4 needs about 44 GB, and Mixtral 8x22B at Q4 needs about 80 GB. None of these fit on any single consumer GPU. The solution is pooling VRAM across multiple GPUs. Two RTX 3090s provide 48 GB of effective VRAM for about $1500-1800 on the used market. Three RTX 3090s give you 72 GB for about $2500. This is dramatically cheaper than a single workstation GPU with equivalent VRAM: the RTX 6000 Ada with 48 GB costs $6800 new. Multi-GPU is not about speed (scaling is sublinear) — it is about capacity. If the model does not fit on one card, adding a second card is the only way to run it locally without dropping to unusably low quantization levels or switching to Apple Silicon.

How Tensor Parallelism Works in llama.cpp

llama.cpp implements tensor parallelism to split model layers across multiple GPUs. There are two main strategies. Row-wise splitting divides each layer's weight matrices across GPUs, with each GPU computing a portion of the output. This requires communication between GPUs for every layer during every token generation step. Layer-wise splitting (the simpler approach) assigns entire consecutive layers to different GPUs. GPU 0 handles layers 0-40, GPU 1 handles layers 41-80, etc. The output of each GPU is the input to the next, so data flows sequentially through the GPUs. This means only one GPU is active at a time during generation — you get zero speed benefit from the second GPU. The upside is simpler setup and no requirement for fast interconnects. llama.cpp defaults to layer splitting, which is why multi-GPU setups rarely show speed improvements over single GPUs. With the --split-mode row flag, llama.cpp enables row-wise tensor parallelism that can provide modest speed gains (10-30%) by keeping both GPUs active simultaneously. However, row splitting requires more inter-GPU communication bandwidth, making PCIe speed more relevant.

# Default layer splitting (no speed gain, but pools VRAM)
./llama-cli -m llama-3.3-70b-Q4_K_M.gguf \
  -ngl 999 -c 8192

# Row-wise splitting for speed improvement
./llama-cli -m llama-3.3-70b-Q4_K_M.gguf \
  -ngl 999 -c 8192 \
  --split-mode row --tensor-split 24,24

# Specify which GPUs to use
CUDA_VISIBLE_DEVICES=0,1 ./llama-cli -m model.gguf -ngl 999

Hardware Requirements: Motherboard, PSU, and Cooling

The physical constraints of running multiple GPUs are often the hardest part. Motherboard: you need PCIe slots spaced far enough apart to accommodate GPU coolers. Most consumer motherboards have two x16 slots, but the second is often x4 electrically and physically located where a 2.5-slot GPU blocks it. Look for motherboards with at least 3-slot spacing between PCIe x16 slots. Workstation boards (Threadripper, Xeon W) typically have better slot spacing and more PCIe lanes. Power supply: dual RTX 3090s draw roughly 350W each under sustained load, plus 150-200W for the CPU and system — about 900W total. A quality 1200W PSU is the minimum safe choice. Triple 3090s need 1600W+. Seasonic, Corsair, and EVGA are the trusted brands. Cooling: two 350W GPUs in close proximity generate enormous heat. The top card in a dual-GPU configuration typically runs 10-15°C hotter because it intakes air pre-heated by the bottom card. Blower-style cards (common on workstation GPUs like the A6000) exhaust heat out the back of the case and are better suited for multi-GPU. Open-air coolers (standard on consumer cards) dump heat into the case, requiring excellent case airflow. In a dual-GPU setup, position the hotter-running card on the bottom if using PCIe riser cables, or ensure at least one empty slot between cards. An open-air test bench or mining frame eliminates spacing issues entirely and is worth considering for dedicated inference rigs.

NVLink vs PCIe: Does It Matter for Inference?

The short answer: for layer-split inference (the default in llama.cpp), NVLink does not matter at all. Data passes between GPUs only at layer boundaries, which is a tiny amount of data relative to PCIe bandwidth. A PCIe 4.0 x16 link (32 GB/s) handles this trivially, and even PCIe 3.0 x8 (8 GB/s) is sufficient. For row-wise tensor parallelism, where GPUs communicate continuously during generation, faster interconnects matter more. PCIe 4.0 x16 is still generally sufficient because the amount of data exchanged per token is in the megabytes, not gigabytes. NVLink becomes relevant only in two scenarios. First, very large models (405B+) where the inter-GPU communication volume is proportionally larger. Second, training or fine-tuning, where gradients must be synchronized across GPUs after every batch — a completely different communication pattern from inference. NVLink bridges for RTX 3090s are available for about $50-100 and are worth installing if your motherboard supports them, but do not expect a meaningful inference speed improvement. For the RTX 4090 and 5090, NVLink is not supported at all — NVIDIA removed the connector from consumer Ada Lovelace and Blackwell cards. Multi-GPU inference works perfectly fine without it.

Real-World Performance: What to Expect

Multi-GPU performance scaling for inference is sublinear. With layer splitting, two GPUs give you roughly 1.0x the speed of the faster single GPU — because only one GPU computes at a time. The second GPU adds VRAM capacity but no speed. With row splitting (--split-mode row), two identical GPUs typically achieve 1.3-1.5x the speed of a single GPU. Three GPUs might reach 1.6-2.0x. The scaling is limited by inter-GPU communication overhead. In practice: dual RTX 3090s running Llama 3.3 70B at Q4 achieve 15-22 tok/s with layer splitting. The same model on a single hypothetical 48 GB GPU would achieve about 30-35 tok/s because all layers would be on the same memory bus. An M4 Max with 128 GB unified memory achieves 12-15 tok/s on the same model — slower than dual 3090s but simpler and using dramatically less power. For MoE models like Mixtral 8x22B, multi-GPU is essentially mandatory: the 141B total parameters require about 80 GB at Q4, needing four RTX 3090s or two A6000s. At this scale, expect 5-12 tok/s depending on GPU count and interconnect.

Software Configuration: Step by Step

Setting up multi-GPU inference in llama.cpp requires a few key flags. First, ensure nvidia-smi shows all GPUs. Then, use CUDA_VISIBLE_DEVICES=0,1 to specify which GPUs llama.cpp should use. The -ngl flag should be set high enough to offload all layers (use -ngl 999 to attempt all layers). For layer splitting (default), no additional flags are needed — llama.cpp automatically distributes layers across visible GPUs. You can control the split with --tensor-split: the flag takes space-separated values in GB, indicating how much VRAM each GPU should use for model weights. For row splitting, add --split-mode row. If a model overflows VRAM, llama.cpp will attempt to offload to system RAM, which kills performance. Set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 to allow CUDA managed memory as overflow instead of crashing. Ollama handles multi-GPU automatically with detection and distribution, but gives you less control over the split. LM Studio does not support multi-GPU at all as of mid-2026. ExLlamaV2 (TabbyAPI) supports multi-GPU with better performance scaling than llama.cpp for some configurations, but it is NVIDIA-only and supports fewer model architectures.

# Step 1: Verify GPUs visible
nvidia-smi

# Step 2: Run with auto layer distribution
CUDA_VISIBLE_DEVICES=0,1 ./llama-cli \
  -m llama-3.3-70b-Q4_K_M.gguf \
  -ngl 999 -c 8192

# Step 3: Manual tensor split (24 GB per GPU for dual 3090s)
CUDA_VISIBLE_DEVICES=0,1 ./llama-cli \
  -m llama-3.3-70b-Q4_K_M.gguf \
  -ngl 999 -c 8192 \
  --tensor-split 24,24

# Step 4: With row splitting for speed
CUDA_VISIBLE_DEVICES=0,1 ./llama-cli \
  -m llama-3.3-70b-Q4_K_M.gguf \
  -ngl 999 -c 8192 \
  --split-mode row --tensor-split 24,24

# Step 5: Prevent OOM with unified memory
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./llama-cli \
  -m llama-3.3-70b-Q4_K_M.gguf -ngl 999 -c 8192

Cost-Benefit Analysis: Multi-GPU vs Alternatives

The economics of multi-GPU setups are compelling compared to single workstation cards. Dual RTX 3090s (48 GB total): roughly $1500-1800 used. A single RTX 6000 Ada (48 GB): roughly $6800 new, rarely available used. The dual 3090s deliver the same VRAM at roughly 25% of the cost. The trade-offs are power consumption (700W vs 300W), physical space, cooling complexity, and the lack of ECC memory. Triple RTX 3090s (72 GB total): roughly $2500 used. This compares to a single RTX A100 80GB ($10,000+ used) — again, roughly 25% of the cost. Compared to Apple Silicon, multi-GPU setups offer more performance per dollar at the cost of complexity and power. An M4 Max MacBook Pro with 128 GB costs roughly $4000 and runs 70B models at 15-25 tok/s silently at under 100W. Dual 3090s cost about $1800 (plus the rest of the PC, roughly $2500 total) and run the same model at 15-22 tok/s, consuming 700W with significant fan noise. For dedicated inference servers running 24/7, the Mac's power efficiency matters. For a workstation that also games or does other GPU work, the NVIDIA multi-GPU setup is more versatile.