Getting Started with Local LLMs (2026): The Complete Beginner's Guide

What Is a Local LLM and Why Run One?

A local LLM (large language model) is an AI chatbot that runs on your own computer instead of on a company's servers. Think of ChatGPT or Claude as a restaurant: you order, and the meal arrives from a kitchen you never see. A local model is cooking at home. Setup takes some effort and your stove limits the menu, but nothing leaves your house, and once a model is downloaded it works without internet.

Do You Have the Right Hardware? A Quick Check

The number that matters most is VRAM, the memory built into your graphics card. Picture a workbench: the model has to be laid out on it to be used, so a bigger bench holds a bigger model. If it does not fit, the overflow spills onto your regular RAM, a slower storage room down the hall, and everything slows down. With no graphics card, RAM is all you have. Check yours below, then find your size on the ladder.

1. Windows: open Task Manager, go to Performance, click GPU, and read Dedicated GPU memory.
2. Mac: Apple menu, About This Mac, then read Memory. Apple Silicon shares one pool of memory between the system and the GPU, so that total is what you work with.
3. Linux: run nvidia-smi for an NVIDIA card or rocm-smi for an AMD card.
4. No graphics card? Your system RAM counts instead. That works too, just more slowly.

1. 6 GBAbout 4B parameters. Phi-4 Mini 3.8B needs 3.8 GB in total at Q4_K_M.
2. 8 GB9B models at Q4_K_M. Qwen 3.5 9B needs 6.4 GB in total, with a little room to spare.
3. 12 GBThe 9B model at Q8_0, a higher-quality version (11.0 GB in total, snug), or Gemma 4 12B at Q4_K_M (8.7 GB).
4. 16 GBGemma 4 12B at Q6_K (11.6 GB in total), or the 9B at Q8_0 (11.0 GB) with room for long chats.
5. 24 GB27B models at Q4_K_M, which score higher than 9B models on published benchmarks. Qwen 3.8 27B needs 19.0 GB in total.
6. 48 GB+27B models at near-lossless Q8_0 (Qwen 3.8 27B: 32.8 GB in total), or 70B models squeezed down to Q3_K_M.

Step 1: Install Ollama (The Simplest Way to Start)

Ollama is a free, open-source app (MIT license) that downloads models and runs them for you. It finds your graphics card on its own, so there is nothing to configure. Run the command for your system below, or download the installer from ollama.com/download. On Windows and Mac you also get a small chat window.

# Mac or Linux (in Terminal)
curl -fsSL https://ollama.com/install.sh | sh

# Windows (in PowerShell)
irm https://ollama.com/install.ps1 | iex

# Check it worked (any system)
ollama -v

# Linux only, if the server is not running
ollama serve

• Windows 10 22H2 or newer. NVIDIA owners need driver 551.61 or newer, per Ollama's Windows docs. Ollama then runs in the background.
• macOS 14 (Sonoma) or newer. Apple Silicon (M1 and later) uses the GPU; Intel Macs run on the CPU only.
• Linux: if chatting fails with a connection error, start the server with ollama serve.

Step 2: Run Your First Model

With Ollama installed, running a model is one command. The first time, Ollama downloads the model (several GB, so allow a few minutes) and opens a chat prompt. Find your machine in the table, copy its command, and press Enter. Type a question and press Enter again. Type /bye to leave.

# Your first model
ollama run qwen3.5:9b

# Smaller, for CPU-only PCs
ollama run phi4-mini:3.8b

# Installed models / running now
ollama list
ollama ps

# Delete a model to free disk space
ollama rm qwen3.5:9b

• More room? On a 12 GB or 16 GB card, ollama run qwen3.5:9b-q8_0 is a higher-quality version of the same model (11.0 GB in total), and ollama run gemma4:12b is a different model (8.7 GB in total). A 16 GB card has far more room left over for long chats.
• On an 8 GB card or a 16 GB Mac, nothing bigger fits comfortably. Keep chats short, and on a Mac close big apps first, because macOS needs its share.
• With no graphics card, ollama run qwen3.5:9b also works at about 7 tok/s, but it thinks out loud first, so the smaller model feels much quicker.

• When the download finishes, the prompt reads >>> Send a message (/? for help). If you see it, you are running a local LLM.
• Qwen 3.5 is a 'thinking' model, so it may write out its reasoning first. That is normal. Type /set nothink to switch it off, and /set think to bring it back.

Step 3: Check Your Speed and Where It Is Running

Two quick checks show whether everything works. First, with a model loaded, run ollama ps in a second terminal. In the PROCESSOR column, 100% GPU is what you want; 100% CPU means your graphics card is not being used, and a split such as 48%/52% CPU/GPU means the model spilled into RAM. Second, type /set verbose in the chat and ask something: eval rate, printed after each reply, is your speed in tokens per second.

# In a second terminal, while a model is loaded
ollama ps

# Inside the chat, to see speed after each reply
/set verbose

• A token is a piece of a word, about three quarters of a word on average (LM Studio's docs use the same ratio). At 5 tokens per second that is about 225 words a minute, roughly reading pace.
• Rule of thumb: under 5 tokens per second feels slow, 5 to 15 is comfortable for chat, and 30 or more streams faster than you can read.

Understanding the Basics: Models, Parameters, and Quantization

Four words come up constantly, and each has an everyday picture. After them, the bars show how big the same model is at each quantization level, as a share of its full 16-bit (FP16) size.

• Model: the AI itself, a big file of learned numbers called weights. In Qwen 3.5 9B, the 9B means 9 billion parameters.
• Parameters: the individual numbers inside the model, like knobs on a mixing desk. More knobs usually means a more capable model and a bigger file.
• Quantization: compression for those numbers, like saving a photo as a smaller JPEG. You lose a little detail and the file shrinks a lot.
• Context: the model's short-term memory for your chat, counted in tokens. Longer chats use more memory. Ollama's docs say it defaults to 4k tokens (about 3,000 words) on cards under 24 GB of VRAM; this page's numbers assume 8k.

Prefer Clicking? Run Local LLMs With LM Studio

LM Studio is a desktop app with a ChatGPT-style window and a built-in model store, so you never open a terminal. It is free to use at home and at work (LM Studio's July 2025 announcement), though the app is closed source. It needs an Apple Silicon Mac on macOS 14 or newer, a Windows PC with AVX2 (16 GB of RAM and 4 GB of VRAM recommended), or Linux on Ubuntu 20.04 or newer.

1. Download LM Studio from lmstudio.ai and install it.
2. Open the Discover tab (Ctrl+2, or Cmd+2 on a Mac) and search for a model, for example Qwen3.5-9B.
3. Pick a Q4_K_M file or better from a well-known publisher such as lmstudio-community or unsloth. Search results also include random re-uploads named 'abliterated' or 'uncensored' that you do not want for a first model.
4. When it finishes, open the Chat tab, open the model loader, choose your model, and start typing.

Next Steps: Where to Go From Here

Pick whichever of these you are curious about next. The r/LocalLLaMA community on Reddit is a good place for news and troubleshooting (search before you post, as its rules ask), and Ollama has its own Discord and r/ollama.

Common Beginner Mistakes (and How to Avoid Them)

Almost everyone hits a few of these in the first week. Here is the list, so you can skip straight past them.

1. Picking a model that is too big. If replies crawl, run ollama ps: a PROCESSOR value like 48%/52% CPU/GPU means the model spilled out of your graphics card. Choose a smaller model or quantization, and check the table above or the CanItRun calculator.
2. Thinking the first reply is broken. A new model takes a few seconds to load, and a thinking model such as Qwen 3.5 then writes out its reasoning before it answers. Wait a moment, or type /set nothink.
3. Forgetting that models stay loaded. Ollama keeps a model in memory for 5 minutes after your last message. To free the memory sooner, run ollama stop followed by the model name.
4. Running two models at once. Ollama loads several only if there is free memory for all of them, so on a consumer card stick to one and check ollama ps.
5. Comparing a local model to the biggest cloud chatbots. A 9B model on your PC is not a frontier model on a datacenter cluster. Compare local models with each other, and treat benchmark scores as a rough guide.
6. Ignoring context length. Ollama's docs say it defaults to a 4k context on cards under 24 GB of VRAM, so long documents get cut off. For one chat, type /set parameter num_ctx 8192. For every chat, use the context slider in Ollama's settings, or start the server with OLLAMA_CONTEXT_LENGTH=8192 ollama serve. More context needs more memory, so recheck ollama ps.
7. Downloading everything. Ollama lists qwen3.5:27b at 17 to 20 GB. Delete models you do not use with ollama rm.