CanItRun Logocanitrun.

Phi-3.5 Mini Instruct

Phi-3.5 Mini Instruct needs roughly 6.2 GB VRAM at Q4_K_M quantization (12.1 GB at FP16). 102 GPUs we track can run it fully in VRAM at 8k context.

102 GPUs run this natively · 2 with CPU offload

Microsoft3.8B params125k contextMITCommercial use ok

Phi-3.5 Mini Instruct is a 3.8B parameter dense model developed by Microsoft. August 2024 3.8B model with 128K context — no GQA means larger KV cache.

To run Phi-3.5 Mini Instruct locally: Q6_K ~4GB — runs on 8GB GPUs. Note: full KV heads increase context memory.

MMLU-Pro 35.6% is strong for sub-4B. HumanEval 62.8%.

VRAM at each quantization

Phi-3.5 Mini Instruct natively supports a longer context window, but the table below is capped at 8k for comparability — KV cache grows linearly with context length.

QuantWeightsKV cacheTotal
FP3215.2 GB3.22 GB20.6 GB
BF167.6 GB3.22 GB12.1 GB
FP167.6 GB3.22 GB12.1 GB
Q8_04.0 GB3.22 GB8.1 GB
Q6_Krec3.1 GB3.22 GB7.1 GB
Q5_K_M2.7 GB3.22 GB6.6 GB
Q4_K_M2.3 GB3.22 GB6.2 GB
Q3_K_M1.8 GB3.22 GB5.7 GB
Q2_K1.4 GB3.22 GB5.2 GB
NVFP4cuda1.9 GB3.22 GB5.7 GB

Shown at 8k context with FP16 KV cache. NVFP4 needs a CUDA GPU to run. Toggle TurboQuant in the calculator to view compressed KV cache numbers.

Benchmarks

MMLU-Pro
47.4
GPQA Diamond
28
IFEval
65.7
MATH
48.5

GPUs that run Phi-3.5 Mini Instruct natively (102)

Plus 2 GPUs that run it with CPU offload (slower)

Notes

No GQA — full KV heads means large KV cache at long context.

Hugging Face ↗Ollama ↗Released 2024-08-21

Compare Phi-3.5 Mini Instruct with other models

How to run Phi-3.5 Mini Instruct locally

816244880160320

Q6_K needs 7.1 GB — fits a single high-end consumer GPU (24 GB).

Ollama

ollama run phi3.5:3.8b

llama.cpp

./llama-cli -m phi-3.5-mini-instruct.Q6_K.gguf -c 4096 -ngl 99

LM Studio: Search for 'Phi 3.5 Mini' in LM Studio. Runs on almost any GPU, but be cautious with long contexts -- the full KV heads create a large cache.

Why this quantization? At 3.8B parameters, Q6_K only costs about 3.5 GB of VRAM for weights, making it trivial to fit on any modern GPU. The key caveat is that Phi-3.5 Mini uses full attention heads (32 KV heads, no GQA), so the KV cache grows faster than typical models at long context. Running at Q6 rather than Q4 preserves the model's strong MMLU-Pro score (47.4) without meaningfully impacting the total VRAM budget.

Who is Phi-3.5 Mini Instruct for?

Developers building edge applications, running LLMs on laptops, or prototyping on budget hardware. The 128K context length is impressive for a model this small, though the lack of GQA means you will hit VRAM limits before filling that context. Great for anyone who needs MIT-licensed inference on minimal hardware.

Best for

  • On-device and edge deployment where GPU memory is very limited
  • Laptop-based AI assistants that run alongside other applications
  • Rapid prototyping and development of LLM-powered features
  • Text classification and simple NLP tasks at high throughput

Not ideal for

  • Utilizing the full 128K context -- the non-GQA architecture makes the KV cache expensive at long sequences
  • Tasks requiring nuanced reasoning or strong math skills -- Phi-4 is a massive step up
  • Multilingual work -- this model is primarily English-focused

Continue reading

vram-guides8 min
Best LLMs You Can Run on 8 GB VRAM (2026)
vram-guides8 min
Best LLMs for 6 GB VRAM (2026)
tutorials9 min
Getting Started with LM Studio: GUI-Based Local LLMs
hardware9 min
Best GPUs Under $500 for LLMs (2026)
tutorials11 min
Getting Started with Local LLMs (2026): The Complete Beginner's Guide

Frequently asked questions

What are the VRAM requirements for Phi-3.5 Mini Instruct?
Phi-3.5 Mini Instruct requires approximately 6.2 GB of VRAM at Q4_K_M quantization, 8.1 GB at Q8, and 12.1 GB at FP16. These numbers assume 8k context window; VRAM scales linearly with context length due to the KV cache.
How many parameters does Phi-3.5 Mini Instruct have?
Phi-3.5 Mini Instruct has 3.8 billion parameters.
How capable is Phi-3.5 Mini Instruct?
Phi-3.5 Mini Instruct has an MMLU-Pro score of 47.4, making it well-suited for lightweight tasks, prototyping, and resource-constrained environments.
Can Phi-3.5 Mini Instruct run on a 16 GB GPU?
Yes. Phi-3.5 Mini Instruct needs 6.2 GB at Q4_K_M, which fits in a 16 GB GPU like the RTX 4080 or RTX 4070 Ti Super.
What is the smallest quantization for Phi-3.5 Mini Instruct that fits in 24 GB of VRAM?
At FP32, Phi-3.5 Mini Instruct needs 20.6 GB — the highest-quality quantization that fits in 24 GB of VRAM.
What GPU do I need to run Phi-3.5 Mini Instruct locally?
A 16 GB GPU is enough. At Q4_K_M, Phi-3.5 Mini Instruct needs 6.2 GB VRAM. Good options: RTX 4080 (16 GB), RTX 4070 Ti Super (16 GB).