CanItRun Logocanitrun.

Nemotron 3.5 Lightning 30B-A3B

Nemotron 3.5 Lightning 30B-A3B needs roughly 28.6 GB VRAM at Q4_K_M quantization (73.8 GB at FP16). 68 GPUs we track can run it fully in VRAM at 8k context.

68 GPUs run this natively · 35 with CPU offload

NVIDIA30B params3B active (MoE)1024k contextOpenMDW-1.1Commercial use ok
Q4_K_M total
28.6 GB
at 8k context
Smallest GPU
32 GB
NVIDIA RTX 5090, at Q4_K_M
KV cache, full context
6.4 GB
6 of 52 layers cache
Inputs
Text
OpenMDW-1.1, released August 2026
Active experts
6 of 128 + 1 shared
~3B active per token
Caching layers
6 of 52
Mamba-2 + MoE layers hold no growing KV cache

Nemotron 3.5 Lightning 30B-A3B is a Mixture of Experts (MoE) model with 30B total parameters but only 3B active per token developed by NVIDIA. Released 11 August 2026 under NVIDIA's own OpenMDW-1.1 license, alongside NeMo Switchyard, a separate open source request-routing tool. It keeps the same 52-layer hybrid stack as Nemotron 3 Nano before it: 26 Mamba-2 layers, 20 MoE layers (128 routed experts, 6 active plus 1 shared per token), and 6 ordinary attention layers, so only 6 of 52 layers keep a KV cache that grows with context. 30B total parameters, 3B active per token, context up to 1,048,576 tokens, text only.

To run Nemotron 3.5 Lightning 30B-A3B locally: Q4_K_M weights are 25.48GB (bartowski's real GGUF; this site uses the real file sizes for this model rather than a generic bits-per-weight estimate), so the standard table's 8k-context total is about 28.6GB: too big for a 24GB card (RTX 4090, RTX 3090) but within reach of a 32GB RTX 5090. On a 24GB card Q3_K_M (19.82GB of weights, about 22.3GB total) is the largest build that fits, and the files barely shrink below Q4: Q2_K is still 18.91GB. The hybrid stack keeps long context cheap in KV terms: filling the full 1,048,576-token window only adds about 6.4GB of cache, versus roughly 55.8GB if every layer used full attention, but at that window even Q2_K totals 28.4GB, so it takes a 32GB card or larger. NVIDIA's own single-GPU claim (1x DGX Spark or 1x H100) checks out at BF16 too: the native checkpoint is 65.85GB of weights and about 73.8GB in total at 8k context, comfortably inside a 128GB DGX Spark. NVFP4, the format NVIDIA shipped first, only runs on Blackwell cards (RTX 5090, DGX Spark, GB200) per this site's own gating: NVIDIA's checkpoint is 21.6GB, about 24.3GB in total at 8k context on an RTX 5090. GGUF builds for llama.cpp and Ollama weren't out at launch (day-one runtimes were vLLM, SGLang, and TensorRT-LLM against the native NVFP4/FP8/BF16 checkpoints), but community GGUFs and an Ollama library entry have since appeared. As a MoE model, inference speed depends on active parameters (3B) rather than total size.

NVIDIA reports MMLU-Pro 81.62 and GPQA Diamond 75.57 (no tools), measured under its own NeMo Evaluator harness. On SWE-bench Verified it scores 52.80, behind GPT-OSS 120B's 62.4, but from a model with roughly a quarter of the total parameters and about three fifths of the active ones. The headline claim is speed, not raw accuracy: up to 4x faster output than similarly-sized models and, per NVIDIA's own PinchBench numbers, 30% faster completion of 10,000 agent tasks than Qwen 3.6 35B-A3B at comparable accuracy.

Three layer types, and only one of them remembers

Nemotron 3.5 Lightning routes every token through one of three layer types across its 52-layer stack: 26 Mamba-2 state-space layers, 20 MoE feed-forward layers (128 routed experts, 6 active plus 1 shared per token), and 6 ordinary attention layers. Only the third kind, the attention layers, stores a key and value for every token it has seen. The strip below groups each kind together for legibility; in the real stack the three interleave throughout, not sitting in three blocks.

Mamba-2state-space, fixed-size recurrent stateMoE FFN128 routed experts, 6 active + 1 shared, no KVFull Attention2 KV heads × 128 dim, GQA
layer 126:20:6 pattern × 1layer 52

6 of 52 layers keep a KV cache that grows with the full context. The other 46 hold a fixed-size recurrent state, so their memory cost does not move as the context window grows.

Mamba-2 layers update a fixed-size state matrix in place, so they cost the same whether the prompt is 2,000 tokens or 2,000,000. MoE layers never look backward at all; they route each token through a handful of experts and move on. Only the 6 attention layers accumulate a cache that grows with context, and NVIDIA scattered them through the stack rather than grouping them, so no single stretch of the network is doing all the remembering.

What a full 1,048,576-token context actually costs

Long context is usually where a local setup falls apart: the KV cache grows with every token fed to it, and on a conventional model it can dwarf the weights long before the advertised window fills up. Capping the cache to 6 of 52 layers is what keeps Nemotron 3.5 Lightning's 1M-token window affordable.

015304560128k512k768k1024k6.4 GBNemotron 3.5 Lightning 30B55.8 GBall-attention 30B
Nemotron 3.5 Lightning 30B-A3B: 6 of 52 layers cacheThe same 30B-A3B if all 52 layers used full attention

KV cache only, at FP16. Weights and activation overhead sit on top of these figures.

Filling the full 1,048,576-token window costs 6.4 GB of KV cache instead of 55.8 GB. Add 25.5 GB of Q4_K_M weights (the real GGUF file) and the full-context total is 35.8 GB, past a 24 GB card's real ceiling and past a 32 GB one's too, so it takes a 48 GB-class card or a unified-memory machine. The all-attention version of the same model would need 91.1 GB at the same context, out of reach for every single consumer GPU this site tracks.

VRAM at each quantization

Numbers here are computed at 8k context. This model's hybrid attention stack means KV cache grows much more slowly than context length, unlike a conventional full-attention model.

QuantWeightsKV cacheTotal
FP32131.7 GB0.05 GB147.6 GB
BF1665.8 GB0.05 GB73.8 GB
FP1665.8 GB0.05 GB73.8 GB
Q8_035.0 GB0.05 GB39.3 GB
Q6_K34.3 GB0.05 GB38.5 GB
Q5_K_M27.0 GB0.05 GB30.3 GB
Q4_K_Mrec25.5 GB0.05 GB28.6 GB
Q3_K_M19.8 GB0.05 GB22.3 GB
Q2_K18.9 GB0.05 GB21.2 GB
NVFP4cuda21.6 GB0.05 GB24.3 GB

KV cache is calculated at 8k context (FP16). Note that NVFP4 only runs on CUDA GPUs. Turn on TurboQuant in the calculator above for lower KV cache estimates.

Quantization × context: where it fits

The table above holds context fixed at 8k. Add context as a second axis and the hybrid stack's story becomes visible: most of the movement between columns comes from each quantization's fixed weight size, not from a KV cache that scales with tokens the way a conventional model's would.

Total VRAM needed by quantization and context length, with the GPU class each figure requires.
Quant8k ctx32k ctx128k ctx1024k ctx
Q8_039.3 GB48 GB card39.4 GB48 GB card40.1 GB48 GB card46.4 GB48 GB card
Q6_K38.5 GB48 GB card38.6 GB48 GB card39.3 GB48 GB card45.6 GB48 GB card
Q5_K_M30.3 GB32 GB card30.4 GB32 GB card31.1 GB32 GB card37.4 GB48 GB card
Q4_K_Mrec28.6 GB32 GB card28.8 GB32 GB card29.4 GB32 GB card35.8 GB48 GB card
Q3_K_M22.3 GB24 GB card22.4 GB24 GB card23.1 GB24 GB card29.4 GB32 GB card
Each cell shows the smallest card that holds it:Consumer GPUWorkstation GPU

Benchmarks

MMLU-Pro
81.6
GPQA Diamond
75.6
SWE-bench Verified
52.8
Terminal-Bench 2.1
23.5

A 30B model chasing a 117B one

The nearest real comparison for a 3B-active agent model is GPT-OSS 120B, OpenAI's own single-GPU release: 62.4 on SWE-bench Verified against Nemotron 3.5 Lightning's 52.80, a real gap, but from a model with roughly a quarter of the total parameters and about three fifths of the active ones. NVIDIA isn't claiming to beat it here; the pitch is the speed side of that trade, up to 4x faster output per NVIDIA's own benchmarks.

48658210B100B1T10TTotal parameters (log scale)GPT-OSS 120BGLM-4.6 355BGLM-4.7 358BQwen 3.5 35B-A3B (MoE)Nemotron 3.5 Lightning 30B-A3BQwen 3.6 27BUI-Mate 27BNemotron 3 Ultra 550B-A55BMiniMax M3InklingKimi K3
SWE-bench Verified, as reported by each model's own publisher. Scaffolding and harness differ between vendors, so read the gaps as approximate rather than a ranking.

GPUs that run Nemotron 3.5 Lightning 30B-A3B natively (68)

Show 63 more
Plus 35 GPUs that run it with CPU offload (slower)

No GPU? Rent an RTX A6000 from $0.33/hr ↗

Prices as of Oct 13, 2026. Rates change; check RunPod for current pricing. Affiliate link: CanItRun may earn a commission. Doesn't affect the fit calculation above.

Notes

Hybrid Mamba-2 / MoE / Attention stack (128 routed experts, 6 active + 1 shared). Only 6 of 52 layers keep a growing KV cache. Built for high-throughput specialized agent execution rather than frontier chat.

Hugging Face ↗Ollama ↗Released 2026-08-11

Compare Nemotron 3.5 Lightning 30B-A3B with other models

How to run Nemotron 3.5 Lightning 30B-A3B locally

816244880160320

Q4_K_M needs 28.6 GB: needs a workstation or datacenter GPU (48–80 GB).

vLLM (DGX Spark / GB10)

vllm serve --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --reasoning-parser nemotron_v3 \
  --speculative_config.method dspark \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Ollama

ollama run nemotron-3.5-lightning:30b

LM Studio: Not in the 11 August 2026 launch-day set: NVIDIA's day-one runtimes were vLLM, SGLang, and TensorRT-LLM against the native NVFP4/FP8/BF16 checkpoints. GGUF builds from bartowski, Unsloth, and ggml-org have since appeared on Hugging Face (bartowski's card lists LM Studio as compatible and says it needs llama.cpp release b10362 or newer), so search for those repos in LM Studio's model browser. Whether LM Studio's own catalog lists the model is unconfirmed here.

Why this quantization? Q4_K_M puts the weights at 25.48 GB (bartowski's real GGUF; Ollama's default tag is the same 25 GB build, and this site uses the real file sizes for this model). Because only 6 of 52 layers keep a KV cache that actually grows with context, the total barely moves at short context: about 28.6 GB at 8k tokens, which fits a 32 GB card but not a 24 GB one. On a 24 GB card the largest build that fits is Q3_K_M (19.82 GB of weights, about 22.3 GB total); IQ4_XS (18.92 GB) is the other practical pick. Push toward the full 1,048,576-token window and the KV cache adds another 6.4 GB: Q4_K_M totals 35.8 GB there, past even a 32 GB card, and Q2_K (28.4 GB) is what holds the whole window on one. Owners of Blackwell hardware can run the native NVFP4 checkpoint instead, 21.6 GB of weights and about 24.3 GB in total at 8k.

Who is Nemotron 3.5 Lightning 30B-A3B for?

Builders running many parallel or always-on agent instances (cybersecurity triage, code review, terminal automation) who need high tokens/sec per GPU more than frontier-level reasoning, and who have a Blackwell or Hopper card, or a 32 GB consumer GPU such as the RTX 5090 (a 24 GB card is limited to Q3_K_M and below), to run it on.

Best for

  • High-volume, narrow agent tasks inside a larger multi-agent system, the workload NVIDIA built it for
  • Long-context tool use and terminal automation, where the 1,048,576-token window stays cheap thanks to the hybrid stack
  • Post-training on proprietary data and tools, since the OpenMDW-1.1 license allows unrestricted commercial fine-tuning
  • RTX 5090 or DGX Spark owners who want the native NVFP4 checkpoint's speed rather than a GGUF

Not ideal for

  • Anyone who needs the single strongest open-weight model available; NVIDIA's own numbers trail GPT-OSS 120B on SWE-bench Verified
  • 24 GB cards at Q4_K_M: the real Q4_K_M GGUF is 25.48 GB, so llama.cpp, Ollama, and LM Studio users on 24 GB need Q3_K_M or IQ4_XS instead
  • 16 GB GPUs: even the smallest real GGUF (Q2_K, 18.91 GB) is bigger than the card's VRAM, so every build offloads to system RAM
  • Workloads that need independently reproduced benchmark numbers before adopting a model; this released the same day as this page was written

So should you run it?

Nemotron is bigger on disk than its 30B headline suggests: the real Q4_K_M GGUF is 25.5 GB, so at short context it needs about 28.6 GB in total, which fits a 32 GB RTX 5090 (169.2 tokens/second by this site's estimate) but not a 24 GB RTX 4090 or RTX 3090. Q3_K_M (19.8 GB of weights, 22.3 GB total) is the largest build that fits a 24 GB card, at about 152.6 tokens/second on an RTX 4090, fast even before counting NVIDIA's separate 4x speed claim over similarly-sized models. The files barely shrink below Q4: Q2_K is still 18.9 GB. Push the context window out toward the full 1,048,576 tokens and a 32 GB card holds Q3_K_M (29.4 GB) or Q2_K (28.4 GB) but not Q4_K_M (35.8 GB), because the hybrid stack keeps the KV cache small but not the weights. Owners of a Blackwell card (RTX 5090, DGX Spark) can run NVIDIA's native NVFP4 checkpoint instead, 21.6 GB of weights and about 24.3 GB in total at 8k context. This isn't a model built to win a chat leaderboard: NVIDIA's own numbers put it behind GPT-OSS 120B on SWE-bench Verified, and its GGUF and Ollama builds only arrived after the 11 August launch. It's built for the always-on, high-volume side of agent systems, the kind of workload where more throughput per GPU matters more than a few extra benchmark points.

Explore

Browse all guides →Browse all models →Browse all GPUs →

Frequently asked questions

What are the VRAM requirements for Nemotron 3.5 Lightning 30B-A3B?
Nemotron 3.5 Lightning 30B-A3B requires approximately 28.6 GB of VRAM at Q4_K_M quantization, 39.3 GB at Q8, and 73.8 GB at FP16. These numbers assume 8k context window; its hybrid attention stack caches far fewer than all layers, so VRAM grows much slower than linearly with context.
How many parameters does Nemotron 3.5 Lightning 30B-A3B have?
Nemotron 3.5 Lightning 30B-A3B has 30 billion total parameters, but only 3 billion are active per token thanks to its Mixture of Experts (MoE) architecture. This makes inference significantly faster than the total parameter count suggests.
How capable is Nemotron 3.5 Lightning 30B-A3B?
Nemotron 3.5 Lightning 30B-A3B achieves an MMLU-Pro score of 81.62, placing it among the most capable open-weight models available, competitive with frontier systems on general knowledge and reasoning.
Can Nemotron 3.5 Lightning 30B-A3B run on a 16 GB GPU?
No. At Q4_K_M, Nemotron 3.5 Lightning 30B-A3B needs 28.6 GB of VRAM, more than 16 GB. You will need a 32 GB GPU like the RTX 5090.
Can Nemotron 3.5 Lightning 30B-A3B run on a 24 GB GPU?
No. Even at Q4_K_M, Nemotron 3.5 Lightning 30B-A3B needs 28.6 GB. Consider a 32 GB card like the RTX 5090.
What is the smallest quantization for Nemotron 3.5 Lightning 30B-A3B that fits in 24 GB of VRAM?
At Q3_K_M, Nemotron 3.5 Lightning 30B-A3B needs 22.3 GB, the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Nemotron 3.5 Lightning 30B-A3B locally?
A 32 GB GPU is the minimum. At Q4_K_M, Nemotron 3.5 Lightning 30B-A3B needs 28.6 GB VRAM. Good option: RTX 5090 (32 GB).
Can I run Nemotron 3.5 Lightning 30B-A3B in the cloud?
Yes. The cheapest RunPod setup we found that holds Nemotron 3.5 Lightning 30B-A3B entirely in GPU memory at 8k context is 1× RTX A6000 at Q8_0, at roughly 105.1 tokens per second and from $0.33 per hour for the whole pod. Prices as of Oct 13, 2026; RunPod rates change, so check their pricing page for the current figure.
How does Nemotron 3.5 Lightning compare to GPT-OSS 120B?
NVIDIA's own numbers put GPT-OSS 120B ahead on SWE-bench Verified, 62.4 versus Nemotron 3.5 Lightning's 52.80, but GPT-OSS carries 117B total parameters and 5.1B active against Lightning's 30B total and 3B active. NVIDIA isn't positioning Lightning as the smarter model; the claim is throughput, up to 4x faster output for tasks that don't need GPT-OSS's extra reasoning headroom.
Does Nemotron 3.5 Lightning actually run on NVIDIA's own DGX Spark?
Yes, at least at BF16. This site's own calculator puts the native BF16 checkpoint at about 73.8 GB total at 8k context, comfortably inside a 128 GB DGX Spark, which matches NVIDIA's stated single-GPU deployment target of one DGX Spark or one H100.
Why does Nemotron 3.5 Lightning mix Mamba-2, MoE, and attention layers instead of just one?
Each layer type is doing a different job cheaply. Mamba-2 layers carry a fixed-size state that costs the same at any context length, MoE layers route each token through only 6 of 128 experts plus a shared one so most of the 30B parameters sit idle on any given token, and the 6 attention layers are the only place the model actually looks back at everything it has seen. Stacking all three is what lets a 3B-active model hold a 1,048,576-token window without the KV cache taking over the VRAM budget.
Is there a GGUF version of Nemotron 3.5 Lightning for llama.cpp or Ollama?
Yes, since shortly after launch. On 11 August 2026 only native NVFP4, FP8, and BF16 checkpoints existed, for vLLM, SGLang, and TensorRT-LLM. GGUF conversions have since been published by bartowski, Unsloth, and ggml-org on Hugging Face (bartowski's card says they need llama.cpp release b10362 or newer), and Ollama lists the model as nemotron-3.5-lightning with a 1M-context, 25 GB default (ollama run nemotron-3.5-lightning:30b). Mind the file sizes: bartowski's Q4_K_M is 25.48 GB and this site uses the real sizes for this model (the Hugging Face hub lists about 32B parameters against the 30B headline), so a 24 GB card needs Q3_K_M (19.82 GB) or IQ4_XS (18.92 GB) rather than Q4_K_M.