llama.cpp Complete Setup Guide: Build, Configure, and Optimize

What llama.cpp Is and Why You Should Know It

llama.cpp is the foundational open-source inference engine that powers the vast majority of local LLM tools. Ollama wraps it. LM Studio bundles a custom build. GPT4All, Text Generation Web UI, jan.ai, and dozens of other projects use it as their backend. Understanding llama.cpp directly gives you three things that higher-level tools do not: maximum performance (you can set exactly the flags your hardware needs), maximum control (precise VRAM budgeting, KV cache quantization, layer distribution), and the ability to run models on any hardware (CUDA, Metal, ROCm, Vulkan, SYCL, or CPU-only — all from the same codebase). It is a C++ project with no Python dependency for inference, which means it starts fast and runs efficiently even on minimal systems. The GGUF model format was developed by the llama.cpp project and has become the de facto standard for distributing quantized models. If you want to understand why your model runs at 20 tok/s instead of 40, or why you are getting out-of-memory errors despite having apparently enough VRAM, learning llama.cpp's flags will give you the answers. It is not as user-friendly as Ollama or LM Studio, but it is not intimidating either — a handful of flags cover 90% of use cases.

Building from Source: The Right Backend for Your Hardware

The most important build decision is which GPU backend to enable. The default make builds CPU-only, which works everywhere but is 5-20x slower than GPU-accelerated inference. For NVIDIA GPUs, build with CUDA support: make LLAMA_CUDA=1. This requires the CUDA toolkit installed. For Apple Silicon, build with Metal: make LLAMA_METAL=1 (often auto-detected on macOS). For AMD GPUs on Linux, build with ROCm: make LLAMA_HIPBLAS=1. For AMD GPUs on Windows or as a fallback, build with Vulkan: make LLAMA_VULKAN=1. For Intel GPUs, build with SYCL: make LLAMA_SYCL=1. The build produces several binaries: llama-cli (interactive chat), llama-server (HTTP API server), llama-perplexity (model evaluation), llama-bench (performance benchmarking), and the quantize tool for creating your own quantized models. Pre-built binaries are available on the GitHub releases page, but compiling from source ensures the optimizations match your specific CPU and GPU. The build system auto-detects your CPU's instruction set (AVX, AVX2, AVX-512) and enables the appropriate optimizations. On modern CPUs, building from source can yield a 10-20% speed improvement over generic pre-built binaries.

# Clone and build llama.cpp
# Choose ONE GPU backend:

# NVIDIA (CUDA)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Apple Silicon (Metal)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release

# AMD Linux (ROCm)
cmake -B build -DGGML_HIPBLAS=ON
cmake --build build --config Release -j$(nproc)

# AMD/Intel Windows/Linux (Vulkan)
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

Essential Flags: The 10 You Actually Need

llama.cpp has hundreds of flags, but 10 cover the vast majority of use cases. -m specifies the model file path (GGUF format). -p provides a prompt for single-shot generation. -ngl sets the number of GPU layers to offload: use a large number like 999 to offload all layers, or a specific count to leave some on CPU. -c sets context length in tokens (default 512, set to 4096 or higher for practical use). --temp controls randomness (0.0 to 2.0, lower is more deterministic). -n sets how many tokens to generate. -t sets CPU thread count (relevant for prompt processing and CPU-offloaded layers). --cache-type-k and --cache-type-v control KV cache quantization (q8_0 halves cache memory with negligible quality loss). --tensor-split controls VRAM distribution across multiple GPUs (space-separated GB values). -b sets batch size for prompt processing (higher = faster prompt processing but uses more VRAM). For interactive chat, use --chat-template or -cnv (conversation mode). The most important flag for performance is -ngl: if set too low, you waste the GPU. If set too high, the model crashes with out-of-memory. Check GPU memory usage with nvidia-smi while adjusting.

# The 10 essential flags in a typical command
./llama-cli \
  -m model.gguf \              # Model file
  -ngl 999 \                    # GPU layers (999 = all)
  -c 8192 \                     # Context length
  --temp 0.7 \                  # Temperature
  -n 512 \                      # Max tokens to generate
  -t 8 \                        # CPU threads
  --cache-type-k q8_0 \         # KV cache quantization
  --cache-type-v q8_0 \
  -b 512 \                      # Batch size
  -p "Your prompt here"          # Prompt text

Server Mode: Production-Ready API Serving

llama.cpp's server binary (llama-server) provides an HTTP API compatible with OpenAI's Chat Completions format. It is the same used by Ollama under the hood. Key server flags: --port sets the listening port (default 8080), --host sets the bind address (0.0.0.0 for network access, 127.0.0.1 for local only), --api-key enables API key authentication, and --embedding enables the embeddings endpoint for RAG applications. The server supports concurrent requests by queuing them — it processes one generation at a time but can queue multiple requests without dropping them. For production use, consider running llama-server behind a reverse proxy (nginx or Caddy) for SSL termination and request logging. The server also provides a basic web chat interface at the root URL, useful for testing. For single-user setups, the server mode is ideal: start it once on boot and connect from any application that supports OpenAI-compatible APIs. For multi-user setups, be aware that concurrent requests are queued, not parallelized — each user waits for previous generations to finish. If you need true concurrent serving, use vLLM or SGLang instead, though these are more complex to set up.

# Start the server (OpenAI-compatible API)
./llama-server \
  -m model.gguf \
  -ngl 999 \
  -c 8192 \
  --host 127.0.0.1 \
  --port 8080

# Test the server
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.7
  }'

# With API key authentication
./llama-server -m model.gguf --api-key my-secret-key

Performance Tuning: Getting Every Token per Second

Performance tuning in llama.cpp is about finding the bottleneck and adjusting flags accordingly. If prompt processing (time to first token) is slow: increase -b (batch size) — try 512, 1024, or 2048. Larger batches process prompts faster but use more VRAM during prefill. Set -t (threads) to your CPU core count for maximum prompt processing parallelism. If token generation is slow but GPU is not at 100%: check that -ngl is set high enough. If GPU memory is full but generation is still slow, your bottleneck is memory bandwidth (the hardware limit). No flag can fix this — upgrade your GPU or quantize more aggressively. If you are getting out-of-memory errors: reduce -c (context length), use KV cache quantization (--cache-type-k q8_0 --cache-type-v q8_0), or reduce -ngl to leave some layers on CPU. If generation speed is inconsistent (fast then slow): check for thermal throttling. Monitor GPU temperature with nvidia-smi. Sustained inference can push GPUs to their thermal limits, causing clock speed drops. Improve case airflow or undervolt the GPU. For a quick performance benchmark: use llama-bench with your model to see maximum theoretical throughput independent of prompt processing.

# Benchmark maximum generation speed
./llama-bench -m model.gguf -ngl 999

# Benchmark with different batch sizes
./llama-bench -m model.gguf -ngl 999 -b 512
./llama-bench -m model.gguf -ngl 999 -b 1024

# Monitor GPU during generation
watch -n 0.5 nvidia-smi

# Check for thermal throttling
nvidia-smi -q -d TEMPERATURE,CLOCK

Advanced Flags: KV Cache, Flash Attention, and More

Beyond the essentials, several advanced flags can meaningfully improve performance. --flash-attn enables flash attention (faster prompt processing, slightly less VRAM usage), available on CUDA and Metal backends. --no-kv-offload keeps KV cache on CPU instead of GPU — useful when VRAM is tight and you prefer stable generation speed over maximum throughput. --mlock locks model in RAM to prevent swapping — essential on systems with limited RAM. --numa enables NUMA-aware memory allocation on multi-socket systems. --repeat-penalty N discourages repetition (1.0 = off, 1.1 = mild, 1.2+ = aggressive). --top-k and --top-p control token sampling; lower values make output more focused and predictable. --min-p sets a minimum probability threshold for token selection, which can reduce nonsensical output. For very large models on multi-GPU, --split-mode row enables row-wise tensor parallelism for speed improvement over the default layer split. --main-gpu designates the primary GPU. The full list is available via ./llama-cli --help, but these advanced flags cover the cases beyond basic usage.