Multi-GPU Setups for Local LLMs (2026): Split Modes & NVLink

Why Multi-GPU? The VRAM Ceiling Problem

The biggest limit on consumer GPUs for LLM inference is VRAM capacity: the largest card, the RTX 5090, stops at 32 GB. Pooling several cards raises that ceiling. This site's calculator counts a pool as the sum of the cards' VRAM minus a 10% reserve for pooling overhead (a single card only loses 5%), so two 24 GB cards count as 43.2 GB usable, not 48. What a pool buys is capacity, not speed (tok/s below means tokens generated per second). Qwen 3.8 27B, the top-ranked model on this site's board for a pair of RTX 3090s, runs at Q5_K_M on one card (22.13 GB in total, about 35.9 tok/s) and at Q8_0 on two (32.75 GB, about 24.3 tok/s). Those labels are quantization levels: the number is roughly the bits stored per weight, so Q8_0 is near-lossless and Q5_K_M is smaller and slightly lossier. The second card buys the higher quality and room for a longer context, and the decode speed goes down, because the bigger weights are read for every token. The chart shows the other side of the trade by replaying each pooled build on a single card: once a model spills out of VRAM into system RAM, it falls to a few tokens per second. vLLM's own docs give the same advice: if a model fits on a single GPU, distributed inference is probably unnecessary.

How Many GPUs Do You Need? Model Size by Pool Size

Work backwards from the model. The table gives each model's footprint at Q4_K_M and 8K context (weights, the KV cache that holds the conversation so far, and runtime overhead: the same total used everywhere else on this site) and the smallest pool of identical cards that holds it. Each size class leads with its top entry on this site's board. Qwen 3.8 27B leads the board and fits one 24 GB card at Q4_K_M, so a second card gains it nothing at that quant. The 70B class (Qwen 2.5 72B ranks highest in it; Llama 3.3 70B is the usual benchmark yardstick) needs three 24 GB cards or two 32 GB cards for Q4_K_M, and two 24 GB cards only reach Q3_K_M. The 100B-plus mixture-of-experts (MoE) class, led by Qwen 3.5 122B-A10B with Nemotron 3 Super close behind, is where four 24 GB cards or three 32 GB cards start to pay off. Mistral Medium 3.5 (128B) is the only dense model past 100B that eight 24 GB cards can hold. Beyond about 115 GB, which is four 32 GB cards, you are into models like MiniMax M2.7 (229B), which needs 158.53 GB at Q4_K_M. That is eight 24 GB cards by the calculator's count, the point where a Mac Studio or a server is the saner build.

Layer Split vs Tensor Parallelism in llama.cpp

llama.cpp has three ways to spread a model across GPUs, chosen with -sm (split mode). Layer split (-sm layer, the default) gives each card a block of consecutive layers plus its share of the KV cache. A token's activations hop from card to card, so only one card computes at any moment: it pools VRAM well and moves very little data, but adds no speed. Row split (-sm row) cuts every weight matrix across the cards; Johannes Gaessler, who wrote the newer tensor mode, says row mode synchronizes after every operation and is only really worthwhile for old, slow GPUs like the P40. Tensor split (-sm tensor) was merged on April 9, 2026 and is still labelled experimental. It shards every layer across the cards and synchronizes twice per layer, so all cards compute at once. It needs flash attention on and an unquantized KV cache, cannot auto-fit the context (set -c yourself), is strongest on CUDA, and roughly two dozen architectures are listed as broken, so test your model before depending on it. Start with layer split and switch only if you measure a gain.

Hardware Requirements: Motherboard, PSU, and Cooling

The physical constraints are often the hardest part, starting with lanes. Ryzen 9000 desktop chips are listed with 28 PCIe 5.0 lanes, 24 of them usable for devices, and 16 of those serve the graphics slots. Boards that support bifurcation (splitting the x16 graphics slot into two x8 links) give two cards x8/x8, but many boards wire the second long slot at x4 through the chipset, and filling extra M.2 sockets can shrink a slot, so read the manual. NVIDIA's own multi-GPU guide for RTX recommends at least x8/x8 PCIe Gen 5 for two cards and x16/x16 as the best case. Layer split tolerates narrow links, because one activation vector crosses per token; tensor split and batch serving do not. Four cards at full width point to a workstation platform: Threadripper PRO on WRX90 is listed with 128 PCIe 5.0 lanes.

• Power supply: size it for the sum of the cards plus 150 to 250 W for the rest of the PC. Under layer split only one card works at a time, so chat draw sits below the sum, but prompt processing, tensor split and batched serving load every card.
• Power limits are cheap insurance. One published test capped an RTX 3090 from 350 W to 275 W and Qwen 3.6 27B decode went from 32.0 to 32.7 tok/s, consistent with decode being limited by memory bandwidth; it collapsed to 20.6 tok/s at 200 W. That is one card and one blog, so run llama-bench on your own pair before trusting it.
• Cooling: open-air coolers exhaust into the case, so leave a free slot between cards. Puget Systems calls the RTX PRO 6000 Workstation Edition, which uses an RTX 5090-style open cooler, not suitable for most multi-GPU builds and points to the Max-Q version (300 W, dual slot, up to four per workstation in NVIDIA's datasheet).
• Risers and slots: after any change, confirm each card negotiated the generation and width you expect with nvidia-smi -q -d PCIE.

NVLink vs PCIe: Does It Matter for Inference?

For the default layer split, NVLink does not matter, and NVIDIA has mostly dropped it. Among GeForce cards only the RTX 3090 and 3090 Ti list NVLink, as a two-way bridge; NVIDIA's pages list it as No for the RTX 4090 and RTX 5090. The workstation line followed: the RTX A6000 bridges two cards at up to 112 GB/s, but the RTX 6000 Ada has no NVLink (Lenovo's spec table, Puget and ServeTheHome agree), and NVIDIA's RTX PRO 6000 Blackwell Max-Q datasheet lists PCIe 5.0 x16 as its only system interface.

• No bridge, no peer-to-peer: GeForce cards also do not do direct transfers over PCIe. Puget's 2023 test found no peer access on a pair of RTX 4090s, so data stages through host memory. llama.cpp has a GGML_CUDA_P2P switch, but its docs say it needs driver support that is usually limited to workstation and datacenter GPUs, and it can cause crashes.
• For llama.cpp the loss is small. Layer split moves one activation per token, far below the roughly 32 GB/s each way that PCIe 4.0 x16 carries. The tensor-mode benchmarks in the performance section ran on RTX 4090s over plain PCIe 4.0 x16 with no bridge and still showed a 2.27x decode gain on four cards, though the author notes the mode suits fast interconnects better.
• For vLLM the engine decides how much a bridge helps: its docs recommend pipeline over tensor parallelism when GPUs lack NVLink, and secondary write-ups put a 3090 bridge at roughly 40 to 60% more batched throughput, without published methodology.

Mixing GPU Models and Vendors

Layer split does not need matching cards. llama.cpp runs a 3090 and a 4090 together, with -ts set in proportion to each card's usable VRAM (a 24 GB card with a 12 GB card wants 2,1). Per-token time is the sum of each card's share of the work, so a slow card drags only the layers it holds. Tensor modes are pickier: NVIDIA's multi-GPU guide requires identical RTX GPUs, and vLLM's tensor parallelism expects equal VRAM per card. Across vendors, CUDA and ROCm builds cannot share one job, but the Vulkan backend can address NVIDIA, AMD and Intel cards together. It supports layer split; users report row split falling back to layer and tensor split running slowly or crashing on Vulkan. One published hybrid rig, a Tesla V100 with two Radeon AI PRO R9700s on Qwen 3.6 35B-A3B at Q8_0, generated about 62 to 64 tok/s, and its author sums it up as being about fitting a bigger model, not about going faster: tensors pass between vendors through system RAM.

Real-World Performance: What to Expect

Under the default layer split, adding cards does not add decode speed. A May 2024 community benchmark of Llama 3 70B at Q4_K_M in llama.cpp measured 14.71 tok/s on one 48 GB RTX A6000 and 14.44 on four of them, 16.57 on two RTX 3090s and 17.07 on four, 19.22 on two RTX 4090s and 18.94 on eight. More cards, same speed, because one card works at a time. Tensor split changes that: llama.cpp's author measured four RTX 4090s on the same model at 21.25 tok/s with layer split and 48.31 with tensor split (2.27x), and 14.26 against 36.68 at 64K context. Capacity is the other half of the story: on this site's accounting, two RTX 3090s reach Q3_K_M, not Q4_K_M, on a 70B model.

• Gains depend on the model. On two RTX 4090s in the same tables, a 27B dense model at Q8_0 went from 30.61 to 43.03 tok/s (1.41x), but a 35B-A3B MoE at Q8_0 went from 158.23 to 126.58, slower. On four RTX 4090s a 106B-A12B MoE at IQ4_XS gained 17% at short context and 1.86x at 64K.
• Prompt processing often loses. At 512 tokens the 70B on four RTX 4090s processed 1,439 tok/s with layer split and 1,314 with tensor split, so tensor mode trades prompt speed for generation speed at short context. Chat and long agent outputs favor tensor; big prompt ingestion favors layer.
• If the model fits one card, keep it there. This site's model derates a pool's bandwidth by 10% (Qwen 3.8 27B at Q4_K_M: about 40.9 tok/s on one RTX 3090, 32.2 on two), and an InsiderLLM table, with no stated methodology, shows an 8B model slowing from 111.7 to 108.1 tok/s going from one 3090 to two.
• The 70B class on a 48 GB pair: this site puts Llama 3.3 70B at Q4_K_M at 50.75 GB (weights, KV cache and overhead at 8K context) against 43.2 GB usable, so two RTX 3090s reach Q3_K_M (40.72 GB, about 19.5 tok/s) cleanly. People do load the 42.6 GB Q4_K_M file onto two 24 GB cards at short context, as the 2024 benchmark did, but with almost no headroom. A single 48 GB RTX A6000 reaches the same Q3_K_M at about 17.1 tok/s, and an M4 Max with 128 GB runs the 70B at Q4_K_M at about 9.6 tok/s in the calculator's model.

Software Configuration: Step by Step

The engine matters more than any flag. First confirm nvidia-smi lists every card and nvidia-smi topo -m shows how they connect. In llama.cpp, -ngl all offloads every layer (older builds want a number such as 999), -fa on enables flash attention, -ts sets the split proportions and CUDA_VISIBLE_DEVICES picks the cards. If a model overflows VRAM, GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 lets llama.cpp spill into system RAM instead of crashing, at a steep speed cost. If you serve several users at once, vLLM or SGLang with tensor parallelism fits better than llama.cpp; for one user on layer split, llama.cpp is simpler. The commands below cover the common cases, and the table summarizes what each tool does by default.

# Confirm every card is visible, how they connect, and the PCIe link each negotiated
nvidia-smi
nvidia-smi topo -m
nvidia-smi -q -d PCIE

# llama.cpp layer split (default): pools VRAM, one card busy at a time
./llama-server -m model-Q8_0.gguf -ngl all -c 32768 -fa on -sm layer

# llama.cpp tensor split (experimental, best on CUDA): every card computes every layer.
# Needs flash attention, an unquantized KV cache, and an explicit -c.
./llama-server -m model-Q8_0.gguf -ngl all -c 32768 -fa on -sm tensor

# Unequal cards: -ts takes proportions, not gigabytes (24 GB + 12 GB = 2,1)
./llama-server -m model-Q8_0.gguf -ngl all -c 32768 -fa on -sm layer -ts 2,1

# Choose which cards to use
CUDA_VISIBLE_DEVICES=0,1 ./llama-server -m model-Q8_0.gguf -ngl all

# vLLM: tensor parallelism across two cards
vllm serve MODEL --tensor-parallel-size 2

# vLLM: pipeline parallelism (no NVLink, or an odd card count)
vllm serve MODEL --pipeline-parallel-size 2

# SGLang: tensor parallelism across two cards
python -m sglang.launch_server --model-path MODEL --tp 2

Cost-Benefit Analysis: Multi-GPU vs Alternatives

Used RTX 3090 prices have climbed: a September 2026 eBay tracker (GPU Poet) showed roughly $1,200 to $1,450 per card, so a pair costs about $2,400 to $2,900 before the rest of the PC, and four cards about $4,800 to $5,800. The same tracker shows RTX 4090 and 5090 listings far above launch MSRP, so price any plan from live sold listings. Against that, the table runs one 100B-class MoE model through the calculator on four RTX 3090s and on single unified-memory machines (Apple Silicon, Strix Halo and DGX Spark share one memory pool between CPU and GPU). Unified memory is quieter and draws far less power, but it is not faster here: the calculator puts the four-card rig at 82.0 tok/s against 58.4 for the M5 Ultra, 49.6 for the M3 Ultra, 33.2 for the DGX Spark and 27.3 for Strix Halo, since the unified-memory machines either have far less memory bandwidth or pay a larger fixed cost per layer on a mixture-of-experts model. A 96 GB RTX PRO 6000 Blackwell would hold the same model on one card with no pooling at all. Choose a multi-GPU rig when you already own a card or two, need CUDA tooling, fine-tune, or serve several people; choose unified memory for a quiet single-user machine running MoE models.