Getting Started with Ollama: Run Any LLM in One Command

What Is Ollama and Why Use It

Ollama is an open-source tool that makes running large language models locally as simple as running a Docker container. It wraps llama.cpp (the industry-standard C++ inference engine) in a user-friendly package that handles model downloading, GPU detection, memory management, and serving, all behind a clean CLI and REST API. Before Ollama, running a local LLM meant downloading GGUF files from Hugging Face, compiling llama.cpp from source, figuring out the right command-line flags for your hardware, and managing model files manually. Ollama reduces this to a single command. It maintains a library of pre-quantized models that you can pull by name, automatically detects your GPU (CUDA for NVIDIA, Metal for Mac, ROCm for AMD), and manages VRAM allocation so models load without manual tuning. The tool runs as a background service with a REST API, making it easy to integrate with GUI frontends, IDE extensions, and custom applications. Most popular LLM interfaces (Open WebUI, Continue.dev, LM Studio integration layers, and others) support Ollama as a backend. This means you can set up Ollama once and connect multiple tools to it.

Installation on Mac, Linux, and Windows

Installation differs slightly by platform but takes under two minutes on each. On Mac, the simplest method is downloading the app from ollama.ai, which installs as a menu bar application and starts the service automatically. Alternatively, Homebrew users can install via the command line. On Linux, Ollama provides a one-line install script that handles dependency detection and service setup. It installs the binary, creates a systemd service, and configures GPU access automatically. For manual installations on Linux, you can download the binary directly and run it. On Windows, download the installer from ollama.ai. Windows support includes NVIDIA GPU acceleration via CUDA. AMD GPU support on Windows uses the Vulkan backend, which is slower than ROCm on Linux but functional. After installation on any platform, the ollama command should be available in your terminal. The service starts automatically on Mac and Linux. On Windows, it runs as a background process after installation.

# Mac: Install via Homebrew
brew install ollama

# Linux: One-line install script
curl -fsSL https://ollama.ai/install.sh | sh

# Verify installation
ollama --version

# Start the service (if not auto-started)
ollama serve

Running Your First Model

With Ollama running, pulling and running a model is a single command. Ollama downloads the model on first run and caches it locally for subsequent use. The default quantization is typically Q4_K_M, which provides a good balance of quality and memory usage. Ollama's model names follow a convention: the model name, optionally followed by a colon and a tag specifying the variant. Without a tag, Ollama uses the default (usually the latest instruct variant at Q4_K_M). You can specify exact sizes and quantizations via tags. The interactive chat session supports multi-turn conversation out of the box. Type your message and press Enter; the model responds, and you can continue the conversation with full context. Type /bye to exit the session. The model stays loaded in memory for a few minutes after you exit, so re-entering the same model is near-instant.

# Run Llama 3.1 8B (downloads on first run, ~4.7 GB)
ollama run llama3.1:8b

# Run a specific quantization
ollama run llama3.1:8b-instruct-q8_0

# Run a larger model
ollama run qwen2.5:14b

# List available models in the Ollama library
ollama list

# Pull a model without running it
ollama pull gemma3:4b

# Remove a model to free disk space
ollama rm llama3.1:8b

The Ollama Model Library

Ollama maintains a curated library of models at ollama.ai/library. The library includes most major open-weight models: the Llama family, Qwen, Gemma, Phi, Mistral, DeepSeek, and many others. Each model page lists available tags (size variants and quantization levels), download size, and basic usage information. Models are pulled from the library on first use and cached in your home directory (the exact location varies by platform). You can also import custom GGUF files into Ollama using a Modelfile, which is useful for models not in the official library or for using specific quantization levels not offered as pre-built tags. The library is regularly updated with new model releases. When a model you have already downloaded receives an update, running ollama pull with the same tag will download only the changed layers, similar to Docker image updates. One thing to watch: the default tag for popular models changes when new versions release. If you need a specific version for reproducibility, use the full tag rather than relying on the default.

# Browse tags for a specific model
ollama show llama3.1:8b

# See all downloaded models and their sizes
ollama list

# Check disk usage for Ollama models
# Mac: du -sh ~/.ollama/models
# Linux: du -sh /usr/share/ollama/.ollama/models
du -sh ~/.ollama/models

Customizing Models with Modelfiles

Modelfiles are Ollama's configuration files for customizing model behavior, similar to Dockerfiles. They let you set system prompts, adjust generation parameters (temperature, top-p, repeat penalty), configure context length, and create named custom models that combine a base model with your preferred settings. This is particularly useful for creating purpose-specific configurations: a coding assistant with a system prompt tuned for code, a writing assistant with higher temperature, or a concise answerer with a low-token limit. You can also use Modelfiles to import raw GGUF files into Ollama, which is essential for running models or quantization levels not available in the official library. The FROM directive specifies either a library model name or a local GGUF file path. After creating a Modelfile, use ollama create to build it into a named model that you can run like any library model.

# Create a custom Modelfile
cat > Modelfile << 'EOF'
FROM llama3.1:8b

SYSTEM "You are a senior software engineer. Be concise and provide code examples when relevant. Use Python unless another language is specified."

PARAMETER temperature 0.3
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"
EOF

# Build the custom model
ollama create coding-assistant -f Modelfile

# Run your custom model
ollama run coding-assistant

# Import a custom GGUF file
cat > Modelfile << 'EOF'
FROM ./my-model-Q5_K_M.gguf
EOF
ollama create my-model -f Modelfile

Using the Ollama REST API

Ollama exposes a REST API on localhost:11434 that supports both streaming and non-streaming generation. This API is compatible with many tools and can be called from any programming language. The most commonly used endpoints are /api/generate for single completions and /api/chat for multi-turn conversation. Both support streaming (the default), where the response is sent token by token as server-sent events. The API also supports embeddings via /api/embeddings, which is useful for building RAG (retrieval-augmented generation) pipelines. You can list loaded models, check model information, and manage the model lifecycle through API endpoints. Many applications support Ollama's API natively, and the API is also partially compatible with the OpenAI Chat Completions format, making it easy to substitute Ollama for OpenAI in existing applications by changing the base URL. For local development, this API is invaluable: you can build applications against a local LLM and switch to a cloud API for production without changing your application code.

# Generate a completion (streaming)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain recursion in one paragraph"
}'

# Chat with conversation history (non-streaming)
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "stream": false,
  "messages": [
    {"role": "user", "content": "What is VRAM?"}
  ]
}'

# Generate embeddings
curl http://localhost:11434/api/embeddings -d '{
  "model": "llama3.1:8b",
  "prompt": "GPU memory management"
}'

GPU Acceleration Setup

Ollama automatically detects and uses GPU acceleration on supported hardware, but there are cases where manual configuration is needed. On NVIDIA GPUs, CUDA support is built in. If Ollama is not detecting your GPU, verify that NVIDIA drivers are installed (nvidia-smi should show your GPU) and that the CUDA toolkit is accessible. On Mac, Metal acceleration is automatic with Apple Silicon. No configuration is needed. On AMD GPUs, Linux users need ROCm installed for GPU acceleration. Ollama's install script handles this on supported distributions. On Windows, AMD cards use the Vulkan backend, which is automatic but slower than ROCm. To verify GPU usage, run a model and check GPU utilization. On NVIDIA, nvidia-smi should show Ollama's process using GPU memory. On Mac, Activity Monitor's GPU tab should show activity. If GPU acceleration is not working, you will notice dramatically slower generation (1-3 tokens per second instead of 20-50). Common causes include missing drivers, incompatible CUDA versions, or insufficient VRAM (which causes Ollama to fall back to CPU). For multi-GPU NVIDIA systems, Ollama distributes layers across available GPUs automatically. You can control which GPUs are used with the CUDA_VISIBLE_DEVICES environment variable.

# Check if NVIDIA GPU is detected
nvidia-smi

# Verify Ollama is using GPU (run while a model is loaded)
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv

# Force specific GPUs (NVIDIA multi-GPU)
CUDA_VISIBLE_DEVICES=0,1 ollama serve

# Check Ollama logs for GPU detection
# Mac: cat ~/.ollama/logs/server.log
# Linux: journalctl -u ollama

Common Issues and Troubleshooting

The most common Ollama issue is out-of-memory errors when loading a model that exceeds GPU VRAM. The error message is usually explicit: it will mention VRAM or CUDA memory. The fix is either using a smaller model, a more aggressive quantization (e.g., Q3_K_M instead of Q4_K_M), or reducing context length. If a model loads but generation is extremely slow (1-3 tokens per second), GPU acceleration is likely not active, and the model is running on CPU. Check your GPU drivers and verify with nvidia-smi or Activity Monitor. If Ollama refuses to start, check if another instance is already running (it binds to port 11434 exclusively). Kill existing instances before starting a new one. On Linux, check the systemd service status. Model download failures are usually network issues. Ollama supports resuming downloads, so rerunning the pull command picks up where it left off. If a model file is corrupted, delete it from the models directory and pull again. For Mac users: if you see excessive memory pressure or your system becomes unresponsive after loading a large model, the model is too large for your available memory. Close other applications or choose a smaller model. macOS does not crash cleanly when memory is exhausted; it slows to a crawl instead.