CanItRun Logocanitrun.

Nemotron 3.5 Lightning 30B-A3B

Nemotron 3.5 Lightning 30B-A3B needs roughly 20.5 GB VRAM at Q4_K_M quantization (67.3 GB at FP16). 78 GPUs we track can run it fully in VRAM at 8k context.

78 GPUs run this natively · 21 with CPU offload

NVIDIA30B params3B active (MoE)1024k contextOpenMDW-1.1Commercial use ok
Q4_K_M total
20.5 GB
at 8k context
Smallest GPU
24 GB
NVIDIA RTX 4090, at Q4_K_M
KV cache, full context
6.4 GB
6 of 52 layers cache
Inputs
Text
OpenMDW-1.1, released August 2026
Active experts
6 of 128 + 1 shared
~3B active per token
Caching layers
6 of 52
Mamba-2 + MoE layers hold no growing KV cache

Nemotron 3.5 Lightning 30B-A3B is a Mixture of Experts (MoE) model with 30B total parameters but only 3B active per token developed by NVIDIA. Released 11 August 2026 under NVIDIA's own OpenMDW-1.1 license, alongside NeMo Switchyard, a separate open source request-routing tool. It keeps the same 52-layer hybrid stack as Nemotron 3 Nano before it: 26 Mamba-2 layers, 20 MoE layers (128 routed experts, 6 active plus 1 shared per token), and 6 ordinary attention layers, so only 6 of 52 layers keep a KV cache that grows with context. 30B total parameters, 3B active per token, context up to 1,048,576 tokens, text only.

To run Nemotron 3.5 Lightning 30B-A3B locally: Q4_K_M weights land at roughly 18.3GB, so the standard table's 8k-context total is about 20.5GB, within reach of any 24GB card (RTX 4090, RTX 3090, RTX 5090). The hybrid stack keeps long context cheap: filling the full 1,048,576-token window only adds about 6.4GB of KV cache, versus roughly 55.8GB if every layer used full attention, so a 24GB card still holds Q2_K at the full window. NVIDIA's own single-GPU claim (1x DGX Spark or 1x H100) checks out at BF16 too: the native checkpoint totals about 67.3GB at 8k context, comfortably inside a 128GB DGX Spark. NVFP4, the format NVIDIA shipped first, only runs on Blackwell cards (RTX 5090, DGX Spark, GB200) per this site's own gating, but it's small: about 16.9GB total at 8k context on an RTX 5090. GGUF builds for llama.cpp and Ollama weren't out at launch; day-one runtimes are vLLM, SGLang, and TensorRT-LLM against the native NVFP4/FP8/BF16 checkpoints. As a MoE model, inference speed depends on active parameters (3B) rather than total size.

NVIDIA reports MMLU-Pro 81.62 and GPQA Diamond 75.57 (no tools), measured under its own NeMo Evaluator harness. On SWE-bench Verified it scores 52.80, behind GPT-OSS 120B's 62.4, but from a model with roughly a quarter of the total parameters and about three fifths of the active ones. The headline claim is speed, not raw accuracy: up to 4x faster output than similarly-sized models and, per NVIDIA's own PinchBench numbers, 30% faster completion of 10,000 agent tasks than Qwen 3.6 35B at comparable accuracy.

Three layer types, and only one of them remembers

Nemotron 3.5 Lightning routes every token through one of three layer types across its 52-layer stack: 26 Mamba-2 state-space layers, 20 MoE feed-forward layers (128 routed experts, 6 active plus 1 shared per token), and 6 ordinary attention layers. Only the third kind, the attention layers, stores a key and value for every token it has seen. The strip below groups each kind together for legibility; in the real stack the three interleave throughout, not sitting in three blocks.

Mamba-2state-space, fixed-size recurrent stateMoE FFN128 routed experts, 6 active + 1 shared, no KVFull Attention2 KV heads × 128 dim, GQA
layer 126:20:6 pattern × 1layer 52

6 of 52 layers keep a KV cache that grows with the full context. The other 46 hold a fixed-size recurrent state, so their memory cost does not move as the context window grows.

Mamba-2 layers update a fixed-size state matrix in place, so they cost the same whether the prompt is 2,000 tokens or 2,000,000. MoE layers never look backward at all; they route each token through a handful of experts and move on. Only the 6 attention layers accumulate a cache that grows with context, and NVIDIA scattered them through the stack rather than grouping them, so no single stretch of the network is doing all the remembering.

What a full 1,048,576-token context actually costs

Long context is usually where a local setup falls apart: the KV cache grows with every token fed to it, and on a conventional model it can dwarf the weights long before the advertised window fills up. Capping the cache to 6 of 52 layers is what keeps Nemotron 3.5 Lightning's 1M-token window affordable.

015304560128k256k512k768k1024k6.4 GBNemotron 3.5 Lightning 30B55.8 GBall-attention 30B
Nemotron 3.5 Lightning 30B-A3B — 6 of 52 layers cacheThe same 30B-A3B if all 52 layers used full attention

KV cache only, at FP16. Weights and activation overhead sit on top of these figures.

Filling the full 1,048,576-token window costs 6.4 GB of KV cache instead of 55.8 GB. Add 18.3 GB of Q4_K_M weights and the full-context total is 27.7 GB, just past a 24 GB card's real ceiling but well inside a 32 GB one. The all-attention version of the same model would need 83.0 GB at the same context, out of reach for every single consumer GPU this site tracks.

VRAM at each quantization

Numbers here are computed at 8k context. This model's hybrid attention stack means KV cache grows much more slowly than context length, unlike a conventional full-attention model.

QuantWeightsKV cacheTotal
FP32120.0 GB0.05 GB134.5 GB
BF1660.0 GB0.05 GB67.3 GB
FP1660.0 GB0.05 GB67.3 GB
Q8_031.9 GB0.05 GB35.8 GB
Q6_K24.6 GB0.05 GB27.6 GB
Q5_K_M21.4 GB0.05 GB24.0 GB
Q4_K_Mrec18.3 GB0.05 GB20.5 GB
Q3_K_M14.4 GB0.05 GB16.2 GB
Q2_K11.4 GB0.05 GB12.9 GB
NVFP4cuda15.0 GB0.05 GB16.9 GB

KV cache is calculated at 8k context (FP16). Note that NVFP4 only runs on CUDA GPUs. Turn on TurboQuant in the calculator above for lower KV cache estimates.

Quantization × context: where it fits

The table above holds context fixed at 8k. Add context as a second axis and the hybrid stack's story becomes visible: most of the movement between columns comes from each quantization's fixed weight size, not from a KV cache that scales with tokens the way a conventional model's would.

Total VRAM needed by quantization and context length, with the GPU class each figure requires.
Quant8k ctx32k ctx128k ctx1024k ctx
Q8_035.8 GB48 GB card35.9 GB48 GB card36.6 GB48 GB card42.9 GB48 GB card
Q6_K27.6 GB32 GB card27.8 GB32 GB card28.5 GB32 GB card34.8 GB48 GB card
Q5_K_M24.0 GB24 GB card24.1 GB32 GB card24.8 GB32 GB card31.1 GB32 GB card
Q4_K_Mrec20.5 GB24 GB card20.7 GB24 GB card21.4 GB24 GB card27.7 GB32 GB card
Q3_K_M16.2 GB24 GB card16.4 GB24 GB card17.1 GB24 GB card23.4 GB24 GB card
Each cell shows the smallest card that holds it:Consumer GPUWorkstation GPU

Benchmarks

MMLU-Pro
81.62
GPQA Diamond
75.57
SWE-bench Verified
52.8
Terminal-Bench 2.1
23.46

A 30B model chasing a 117B one

The nearest real comparison for a 3B-active agent model is GPT-OSS 120B, OpenAI's own single-GPU release: 62.4 on SWE-bench Verified against Nemotron 3.5 Lightning's 52.80, a real gap, but from a model with roughly a quarter of the total parameters and about three fifths of the active ones. NVIDIA isn't claiming to beat it here; the pitch is the speed side of that trade, up to 4x faster output per NVIDIA's own benchmarks.

48658210B100B1T10TTotal parameters (log scale)GPT-OSS 120BGLM-4.6 355BGLM-4.7 358BQwen 3.5 35B-A3B (MoE)Nemotron 3.5 Lightning 30B-A3BQwen 3.6 27BMistral Medium 3.5 128BNemotron 3 Ultra 550B-A55BMiMo V2.5 ProMiniMax M3InklingKimi K3Muse Glimmer 30B
SWE-bench Verified, as reported by each model's own publisher. Scaffolding and harness differ between vendors, so read the gaps as approximate rather than a ranking.

GPUs that run Nemotron 3.5 Lightning 30B-A3B natively (78)

Show 73 more
Plus 21 GPUs that run it with CPU offload (slower)

Notes

Hybrid Mamba-2 / MoE / Attention stack (128 routed experts, 6 active + 1 shared). Only 6 of 52 layers keep a growing KV cache. Built for high-throughput specialized agent execution rather than frontier chat.

Hugging Face ↗Released 2026-08-11

Compare Nemotron 3.5 Lightning 30B-A3B with other models

How to run Nemotron 3.5 Lightning 30B-A3B locally

816244880160320

Q4_K_M needs 20.5 GB — fits a single high-end consumer GPU (24 GB).

vLLM (DGX Spark / GB10)

vllm serve --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --reasoning-parser nemotron_v3 \
  --speculative_config.method dspark \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

LM Studio: Not available at launch. NVIDIA's day-one runtimes are vLLM, SGLang, and TensorRT-LLM against the native NVFP4/FP8/BF16 checkpoints; LM Studio needs a GGUF build that hadn't shipped as of 11 August 2026.

Why this quantization? Q4_K_M puts the weights at roughly 18.3 GB, and because only 6 of 52 layers keep a KV cache that actually grows with context, the total barely moves at short context: about 20.5 GB at 8k tokens, comfortably inside a 24 GB card. Push toward the full 1,048,576-token window and the KV cache adds another 6.4 GB, enough to push Q4_K_M past a 24 GB card's real ceiling; Q2_K (about 20.0 GB at full context) is the quantization that actually holds the whole window on a single 24 GB GPU. Owners of Blackwell hardware can skip the ladder altogether and run the native NVFP4 checkpoint instead, at about 16.9 GB.

Who is Nemotron 3.5 Lightning 30B-A3B for?

Builders running many parallel or always-on agent instances (cybersecurity triage, code review, terminal automation) who need high tokens/sec per GPU more than frontier-level reasoning, and who have a Blackwell or Hopper card, or a 24 GB+ consumer GPU, to run it on.

Best for

  • High-volume, narrow agent tasks inside a larger multi-agent system, the workload NVIDIA built it for
  • Long-context tool use and terminal automation, where the 1,048,576-token window stays cheap thanks to the hybrid stack
  • Post-training on proprietary data and tools, since the OpenMDW-1.1 license allows unrestricted commercial fine-tuning
  • RTX 5090 or DGX Spark owners who want the native NVFP4 checkpoint's speed without waiting for a GGUF build

Not ideal for

  • Anyone who needs the single strongest open-weight model available; NVIDIA's own numbers trail GPT-OSS 120B on SWE-bench Verified
  • llama.cpp, Ollama, or LM Studio users specifically, since no GGUF conversion had shipped as of launch day
  • 16 GB GPUs at anything beyond short context; even Q4_K_M's 8k-context total exceeds a 16 GB card's real ceiling
  • Workloads that need independently reproduced benchmark numbers before adopting a model; this released the same day as this page was written

So should you run it?

On a single 24 GB card (RTX 4090, RTX 3090, RTX 5090) at short context, Q4_K_M needs about 20.5 GB and this site's own RTX 4090 estimate lands well over 100 tokens/second, fast even before counting NVIDIA's separate 4x speed claim over similarly-sized models. Push the context window out toward the full 1,048,576 tokens and Q4_K_M no longer fits a 24 GB card; Q2_K does, at about 20.0 GB, because the hybrid stack keeps the KV cache small even at a million tokens. Owners of a Blackwell card (RTX 5090, DGX Spark) can skip the GGUF ladder entirely and run the native NVFP4 checkpoint instead, at roughly 16.9 GB. This isn't a model built to win a chat leaderboard: NVIDIA's own numbers put it behind GPT-OSS 120B on SWE-bench Verified, and there's no GGUF or Ollama build yet at launch. It's built for the always-on, high-volume side of agent systems, the kind of workload where more throughput per GPU matters more than a few extra benchmark points.

Explore

Browse all guides →Browse all models →Browse all GPUs →

Frequently asked questions

What are the VRAM requirements for Nemotron 3.5 Lightning 30B-A3B?
Nemotron 3.5 Lightning 30B-A3B requires approximately 20.5 GB of VRAM at Q4_K_M quantization, 35.8 GB at Q8, and 67.3 GB at FP16. These numbers assume 8k context window; its hybrid attention stack caches far fewer than all layers, so VRAM grows much slower than linearly with context.
How many parameters does Nemotron 3.5 Lightning 30B-A3B have?
Nemotron 3.5 Lightning 30B-A3B has 30 billion total parameters, but only 3 billion are active per token thanks to its Mixture of Experts (MoE) architecture. This makes inference significantly faster than the total parameter count suggests.
How capable is Nemotron 3.5 Lightning 30B-A3B?
Nemotron 3.5 Lightning 30B-A3B achieves an MMLU-Pro score of 81.62, placing it among the most capable open-weight models available — competitive with frontier systems on general knowledge and reasoning.
Can Nemotron 3.5 Lightning 30B-A3B run on a 16 GB GPU?
No. At Q4_K_M, Nemotron 3.5 Lightning 30B-A3B needs 20.5 GB of VRAM — more than 16 GB. You will need a 24 GB GPU like the RTX 4090 or RTX 3090.
Can Nemotron 3.5 Lightning 30B-A3B run on a 24 GB GPU?
Yes. Nemotron 3.5 Lightning 30B-A3B fits in a 24 GB GPU at Q4_K_M, requiring 20.5 GB VRAM. GPUs with 24 GB include the RTX 4090, RTX 3090, and RTX 3090 Ti.
What is the smallest quantization for Nemotron 3.5 Lightning 30B-A3B that fits in 24 GB of VRAM?
At NVFP4, Nemotron 3.5 Lightning 30B-A3B needs 16.9 GB — the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Nemotron 3.5 Lightning 30B-A3B locally?
A 24 GB GPU is the minimum. At Q4_K_M, Nemotron 3.5 Lightning 30B-A3B needs 20.5 GB VRAM. Good options: RTX 4090 (24 GB), RTX 3090 (24 GB).
How does Nemotron 3.5 Lightning compare to GPT-OSS 120B?
NVIDIA's own numbers put GPT-OSS 120B ahead on SWE-bench Verified, 62.4 versus Nemotron 3.5 Lightning's 52.80, but GPT-OSS carries 117B total parameters and 5.1B active against Lightning's 30B total and 3B active. NVIDIA isn't positioning Lightning as the smarter model; the claim is throughput, up to 4x faster output for tasks that don't need GPT-OSS's extra reasoning headroom.
Does Nemotron 3.5 Lightning actually run on NVIDIA's own DGX Spark?
Yes, at least at BF16. This site's own calculator puts the native BF16 checkpoint at about 67.3 GB total at 8k context, comfortably inside a 128 GB DGX Spark, which matches NVIDIA's stated single-GPU deployment target of one DGX Spark or one H100.
Why does Nemotron 3.5 Lightning mix Mamba-2, MoE, and attention layers instead of just one?
Each layer type is doing a different job cheaply. Mamba-2 layers carry a fixed-size state that costs the same at any context length, MoE layers route each token through only 6 of 128 experts plus a shared one so most of the 30B parameters sit idle on any given token, and the 6 attention layers are the only place the model actually looks back at everything it has seen. Stacking all three is what lets a 3B-active model hold a 1,048,576-token window without the KV cache taking over the VRAM budget.
Is there a GGUF version of Nemotron 3.5 Lightning for llama.cpp or Ollama?
Not at launch. Nemotron 3.5 Lightning shipped on 11 August 2026 with native NVFP4, FP8, and BF16 checkpoints for vLLM, SGLang, and TensorRT-LLM; NVIDIA's own materials list llama.cpp, Ollama, and Unsloth as supported runtimes, but no GGUF conversion was published on day one. Check the model's Hugging Face repo for whether one has landed since.