GPT-OSS 120B
GPT-OSS 120B needs roughly 80.2 GB VRAM at Q4_K_M quantization (262.4 GB at FP16). 24 GPUs we track can run it fully in VRAM at 8k context.
24 GPUs run this natively · 5 with CPU offload
- Q3_K_M total
- 70.5 GB
- at 8k context
- Smallest GPU
- 80 GB
- NVIDIA H100 80GB, at Q3_K_M
- KV cache, full context
- 4.8 GB
- 18 of 36 layers grow with context, 18 capped at 128 tokens
- Inputs
- Text
- Apache 2.0, released August 2025
- MoE routing
- 4 of 128 experts
- 5.1B of 117B active per token
- Weight floor
- ~63 GB
- same across nearly every quant
GPT-OSS 120B is a Mixture of Experts (MoE) model with 117B total parameters but only 5.1B active per token developed by OpenAI. Released 5 August 2025, 117B MoE with 5.1B active (128 experts, top-4 routed per token). Every layer alternates a 128-token sliding-window block with a full-attention block, and each attention head carries a learned 'sink' bias so it can attend to nothing instead of a real token. Apache 2.0.
To run GPT-OSS 120B locally: OpenAI never released the MoE expert weights (~90% of total params) above native MXFP4: every real GGUF build, from Q2_K up to BF16, keeps those tensors at MXFP4 and only touches the small attention/embedding remainder. Real download sizes cluster 62.6-65.4GB regardless of nominal quant (bartowski/openai_gpt-oss-120b-GGUF), so the honest floor is a single 80GB GPU, exactly matching OpenAI's own single-H100 claim, rather than a wide Q2-to-FP16 range. As a MoE model, inference speed depends on active parameters (5.1B) rather than total size.
GPQA Diamond 80.1% and SWE-bench Verified 62.4% at OpenAI's high reasoning-effort setting, near-parity with o4-mini. Reasoning effort (low/medium/high) is configurable at request time and trades latency for accuracy: SWE-bench Verified alone runs 47.9/52.6/62.4 across the three tiers.
One sliding-window layer for every full-attention layer
GPT-OSS 120B is a mixture of experts: 128 experts sit in the weights, but a router activates only 4 of them per token, which combined with the always-on attention/embedding layers is what keeps 5.1B of the 117B total parameters active per token. Separately, its 36 attention layers alternate between two attention shapes rather than using one throughout: a 128-token sliding-window layer, then a full-attention layer, repeated eighteen times. A sliding-window layer can only look back 128 tokens, however long the conversation gets; a full-attention layer can look all the way back to the first token. Both keep a real KV cache; the difference is how far back each layer is allowed to look. Every layer, sliding or full, also carries a learned per-head "attention sink" bias in the softmax denominator, letting a head effectively attend to nothing instead of a real token, a compute-only detail with no memory cost of its own.
18 of 36 layers keep a KV cache that grows with the full context. 18 more cap their cache at a fixed 128-token window, no matter how long the context gets.
Once a conversation passes 128 tokens (often within the first exchange), the 18 sliding-window layers stop growing; only the 18 full-attention layers keep accumulating. That very nearly halves the KV cache at the full 131,072-token context: 4.84 GB instead of 9.66 GB for an all-full-attention version of the same stack. It's real, but it's a minor saving next to the bigger story on this page: OpenAI released the MoE expert weights (about 90% of the 117 billion total) only in native MXFP4 (roughly 4.25 bits per weight), and no community requantization meaningfully shrinks them further without wrecking quality. Real downloads, from the smallest GGUF quant up to the largest, land within about 3 GB of each other: 62.6 GB to 65.4 GB, per bartowski's openai_gpt-oss-120b-GGUF repo, which is why the standard quantization table above collapses Q2_K and Q3_K down to the same floor: below it, there is simply nothing left in the expert weights to trade away. Treat any row above that floor as this site's generic ladder overstating a model that, in reality, barely changes size at all.
VRAM at each quantization
Calculated at 8k context. Because only part of this model's stack keeps a growing KV cache, longer sessions cost far less extra VRAM than a full-attention model of the same size.
| Quant | Weights | KV cache | Total |
|---|---|---|---|
| FP32 | 468.0 GB | 0.31 GB | 524.5 GB |
| BF16 | 234.0 GB | 0.31 GB | 262.4 GB |
| FP16 | 234.0 GB | 0.31 GB | 262.4 GB |
| Q8_0 | 124.4 GB | 0.31 GB | 139.6 GB |
| Q6_K | 96.1 GB | 0.31 GB | 107.9 GB |
| Q5_K_M | 83.3 GB | 0.31 GB | 93.6 GB |
| Q4_K_M | 71.3 GB | 0.31 GB | 80.2 GB |
| Q3_K_Mrec | 62.6 GB | 0.31 GB | 70.5 GB |
| Q2_K | 62.6 GB | 0.31 GB | 70.5 GB |
| NVFP4cuda | 62.6 GB | 0.31 GB | 70.5 GB |
KV cache figures assume 8k context at FP16. NVFP4 quantization requires a CUDA-capable GPU. Enable TurboQuant in the calculator to see reduced KV cache estimates.
Benchmarks
GPUs that run GPT-OSS 120B natively (24)
- NVIDIA B300 288GBBF16 · 151.6 t/s
- NVIDIA B200 180GBNVFP4 · 590.5 t/s
- NVIDIA H200 141GBQ6_K · 218.7 t/s
- NVIDIA H100 80GBQ3_K_M · 256.7 t/s
- NVIDIA A100 80GBQ3_K_M · 156.2 t/s
Show 19 more
- NVIDIA RTX Pro 6000NVFP4 · 99.2 t/s
- NVIDIA DGX Spark (128GB)NVFP4 · 20.1 t/s
- AMD Instinct MI300XQ8_0 · 187.5 t/s
- AMD Strix Halo (128GB)Q6_K · 11.7 t/s
- AMD Strix Halo (96GB)Q4_K_M · 15.6 t/s
- Apple M5 Ultra (512GB)BF16 · 28 t/s
- Apple M5 Ultra (256GB)Q8_0 · 52.2 t/s
- Apple M5 Ultra (96GB)Q4_K_M · 90.1 t/s
- Apple M5 Max (128GB)Q6_K · 34.4 t/s
- Apple M4 Max (128GB)Q6_K · 30.6 t/s
- Apple M3 Ultra (512GB)BF16 · 19.1 t/s
- Apple M3 Ultra (256GB)Q8_0 · 35.7 t/s
- Apple M3 Ultra (96GB)Q4_K_M · 61.5 t/s
- Apple M3 Max (128GB)Q6_K · 22.4 t/s
- Apple M3 Max (96GB)Q4_K_M · 22.5 t/s
- Apple M2 Ultra (192GB)Q8_0 · 34.8 t/s
- Apple M2 Max (96GB)Q4_K_M · 30 t/s
- Apple M1 Ultra (128GB)Q6_K · 44.9 t/s
- Intel Data Center GPU Max 1550Q6_K · 149.3 t/s
Plus 5 GPUs that run it with CPU offload (slower)
- NVIDIA L40SQ3_K_M · 9.6 t/s
- NVIDIA RTX A6000Q3_K_M · 9.5 t/s
- NVIDIA RTX 6000 AdaQ3_K_M · 9.7 t/s
- AMD Radeon PRO W7900Q3_K_M · 9.6 t/s
- Intel Data Center GPU Max 1100Q3_K_M · 9.9 t/s
How to run GPT-OSS 120B locally
Q3_K_M needs 70.5 GB: needs a workstation or datacenter GPU (48–80 GB).
llama.cpp
llama-server \
-hf ggml-org/gpt-oss-120b-GGUF \
--ctx-size 0 --jinja \
-ngl 99 -ub 2048 -b 2048 \
--chat-template-kwargs '{"reasoning_effort":"high"}'Ollama
ollama pull gpt-oss:120b && ollama run gpt-oss:120bLM Studio: Search for 'gpt-oss-120b' and take whatever build LM Studio offers: every real download, from Q2_K up to the native MXFP4 file, lands within a few GB of the same ~63 GB, since the MoE expert weights ship only in MXFP4 and don't requantize any smaller. Set 'Reasoning Effort' to low/medium/high in the right-hand panel: it changes both speed and benchmark scores, not just verbosity.
Why this quantization? GPT-OSS 120B doesn't have a meaningful quant ladder to choose from. OpenAI released the MoE expert weights (about 90% of the 117B total) only in native MXFP4 (~4.25 bits/weight), and every community GGUF from Q2_K through BF16 keeps those tensors at MXFP4 and only touches the small attention/embedding remainder, so real downloads cluster 62.6-65.4 GB regardless of the label on the file. There's no lower-VRAM escape hatch: pick the native MXFP4 build (or any GGUF repack of it) and budget for an 80 GB GPU.
Who is GPT-OSS 120B for?
Owners of a single 80 GB-class GPU (H100, A100, MI300X, RTX Pro 6000 Blackwell) or a 128 GB+ unified-memory machine who want the strongest open-weight reasoning model that still runs on one card, without a multi-GPU cluster.
Best for
- Agentic coding and tool-use workflows that need near-frontier reasoning locally
- Configurable reasoning effort (low/medium/high) to trade latency for accuracy per request
- Teams standardizing on Apache 2.0 for commercial local deployment
- Workloads that benefit from OpenAI's own harmony response format and function-calling support
Not ideal for
- Anyone without an 80 GB-class GPU or big unified-memory machine, there is no real quantization that shrinks the ~63 GB floor
- Vision or audio input, GPT-OSS is text-only
- Ultra-low-latency single-turn chat, where the smaller GPT-OSS 20B or a dense model responds faster
So should you run it?
If you already have a single 80 GB GPU (H100, A100, MI300X, or an RTX Pro 6000 Blackwell) or 128 GB+ of unified memory, this is close to the easiest local-LLM sizing decision on the site: real downloads land around 62.6 GB to 65.4 GB no matter which quantization you pick, so there's no real quality-for-size trade to weigh the way there is on a normal model. Grab OpenAI's native MXFP4 build directly, or any GGUF repackaging of it; they're all effectively the same size. Below that tier there's no rescue quantization waiting: the expert weights were never released above 4-bit, so pushing the nominal quant down to Q2_K buys almost nothing. Anyone without an 80 GB-class card should look at the smaller sibling, GPT-OSS 20B, instead: the same MoE design and configurable reasoning effort, comfortably inside 16 GB.
Frequently asked questions
- What are the VRAM requirements for GPT-OSS 120B?
- GPT-OSS 120B requires approximately 80.2 GB of VRAM at Q4_K_M quantization, 139.6 GB at Q8, and 262.4 GB at FP16. These numbers assume 8k context window; its hybrid attention stack caches far fewer than all layers, so VRAM grows much slower than linearly with context.
- How many parameters does GPT-OSS 120B have?
- GPT-OSS 120B has 117 billion total parameters, but only 5.1 billion are active per token thanks to its Mixture of Experts (MoE) architecture. This makes inference significantly faster than the total parameter count suggests.
- How capable is GPT-OSS 120B?
- GPT-OSS 120B achieves an MMLU-Pro score of 80.7, placing it among the most capable open-weight models available, competitive with frontier systems on general knowledge and reasoning.
- Can GPT-OSS 120B run on a 16 GB GPU?
- No. At Q4_K_M, GPT-OSS 120B needs 80.2 GB of VRAM, more than 16 GB. You will need a multi-GPU server.
- Can GPT-OSS 120B run on a 24 GB GPU?
- No. Even at Q4_K_M, GPT-OSS 120B needs 80.2 GB. Consider a multi-GPU server with 81 GB+ of combined VRAM.
- What is the smallest quantization for GPT-OSS 120B that fits in 24 GB of VRAM?
- GPT-OSS 120B cannot fit in 24 GB of VRAM at any standard quantization level. The minimum needed is 70.5 GB at Q2_K.
- What GPU do I need to run GPT-OSS 120B locally?
- You need a multi-GPU server. At Q4_K_M, GPT-OSS 120B needs 80.2 GB VRAM, more than any single consumer GPU. That's roughly 2x 80 GB datacenter GPUs (H100, A100, or similar) pooled together.