How to Benchmark Your Local LLM Setup

Why Benchmark? Your GPU Might Be Underperforming

Most local LLM setups are not running at their full potential. Common issues that benchmarks reveal include: GPU acceleration not enabled (running on CPU at 1/10th the speed), insufficient GPU layers offloaded (only half the model on GPU), thermal throttling (GPU clock dropping under sustained load), inadequate power delivery (GPU not reaching boost clocks), slow system RAM bottlenecking partial offloading, and suboptimal context/batch size settings. A quick benchmark run takes 5 minutes and can reveal whether your 24 GB GPU is performing like a 24 GB GPU or like a 12 GB one. Community benchmark databases (r/LocalLLaMA, llama.cpp discussions, CanItRun) provide reference numbers for common GPU + model combinations. If your RTX 3090 generates 20 tok/s on Llama 3.1 8B when the community average is 80 tok/s, something is wrong. Systematic benchmarking also helps you make informed upgrade decisions: you can quantify exactly how much faster a new GPU or different quantization level would be.

Using llama-bench for Raw Throughput

llama.cpp ships with llama-bench, a dedicated benchmarking tool that measures prompt processing speed (tokens per second during prefill) and token generation speed (tokens per second during decode). It runs multiple iterations and reports averages, giving you a clean throughput measurement. Unlike interactive chat, llama-bench isolates the inference engine from terminal rendering and I/O overhead, so the numbers reflect actual GPU performance. Run it with your model and the -ngl flag set to 999 (all layers on GPU). The output shows: model size, backend, number of layers offloaded, prompt processing speed (in tokens per second), and text generation speed (in tokens per second). For a thorough test, run with different batch sizes (-b 128, 512, 1024) to understand how prompt processing speed scales. For multi-GPU setups, llama-bench can test different GPU counts. For CPU testing, vary the thread count (-t) to find the optimal setting. Save your benchmark results — they serve as a baseline for future comparison.

# Basic throughput benchmark
./llama-bench -m model.gguf -ngl 999

# Test with different batch sizes
./llama-bench -m model.gguf -ngl 999 -b 128
./llama-bench -m model.gguf -ngl 999 -b 512
./llama-bench -m model.gguf -ngl 999 -b 1024

# Test prompt processing separately
./llama-bench -m model.gguf -ngl 999 -p 512

# Multi-GPU benchmark
CUDA_VISIBLE_DEVICES=0,1 ./llama-bench -m model.gguf -ngl 999

Measuring Real-World Speed: Tokens Per Second and TTFT

Raw throughput benchmarks are useful, but real-world experience is shaped by two metrics: tokens per second (TPS) during generation and time-to-first-token (TTFT) — the delay between sending your prompt and seeing the first response token. Ollama reports TPS after each generation: look for 'total duration' and 'tokens generated' in the output. Divide tokens by seconds for your effective TPS. In llama.cpp interactive mode, generation speed is printed after each response. For TTFT, use a stopwatch or script the measurement. A good TPS target depends on use case: 10 tok/s is the minimum for comfortable reading (about reading speed), 20-30 tok/s feels responsive, and 50+ tok/s feels instant. TTFT under 1 second is excellent, under 3 seconds is acceptable, and over 5 seconds is frustrating. TTFT is primarily determined by prompt length (longer prompts take longer to process) and your GPU's compute capability (which parallelizes the prefill). To measure consistently, use the same prompt across different models and configurations. For a DIY benchmark script, use curl against Ollama's API with the --stream flag and measure timestamps.

# Measure real-world speed with Ollama
ollama run llama3.1:8b --verbose
# Look for: "eval rate: XX.XX tokens/s"

# DIY benchmark script for Ollama
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write a 500-word essay on GPU architecture."
}' | tee /dev/null

# Measure TTFT with streaming
# Time from request to first streaming chunk

Measuring Output Quality: Does Quantization Degrade?

Throughput benchmarks tell you about speed. Quality benchmarks tell you whether your quantized model is still thinking clearly. Two approaches: automated benchmarks using standard test suites, and manual A/B comparison. For automated testing, llama.cpp includes llama-perplexity which measures a model's perplexity (how 'surprised' the model is by text — lower is better) on a test corpus. Run it with your model against a standard dataset like WikiText-2. Compare perplexity scores across quantization levels for the same model. A significant jump in perplexity from Q4 to Q3 indicates quality loss. For practical quality testing, use human evaluation: give the same complex prompt (requiring multi-step reasoning, factual recall, and creative writing) to the same model at different quantization levels. Compare outputs side by side without knowing which is which. If you cannot reliably tell Q4 from Q8, you are fine at Q4. If Q3 produces noticeably worse output, avoid it. The community maintains shared prompt sets for this purpose on r/LocalLLaMA. For coding quality, use a standardized benchmark like HumanEval or a personal test set of programming problems you know the answers to.

# Measure perplexity on WikiText
./llama-perplexity -m model.gguf -f wikitext-2-raw.parquet

# Compare perplexity across quantizations
./llama-perplexity -m model-Q4_K_M.gguf -f test.txt
./llama-perplexity -m model-Q5_K_M.gguf -f test.txt
./llama-perplexity -m model-Q8_0.gguf -f test.txt
# Smaller perplexity = better quality

Comparing Against Community Benchmarks

The local LLM community maintains shared performance databases. On r/LocalLLaMA, users regularly post benchmark results for new models and hardware. The CanItRun GPU database includes expected TPS ranges for common GPU + model combinations. To compare your results: find a reported benchmark for your GPU + model + quantization combination, run the same benchmark yourself, and compare. If your results are more than 15% below community numbers, investigate: check GPU clock speeds (thermal throttling?), verify with nvidia-smi that the model fits entirely in VRAM (if it is overflowing to RAM, speed will crater), check CPU thread count during prompt processing, and verify your GPU driver version is current. For Apple Silicon benchmarks, compare against other users with the same chip and memory configuration. Community benchmarks are also useful before purchasing: search for '[GPU name] [model name] tokens per second' on Reddit to find real-world numbers from actual owners. These are often more reliable than vendor marketing claims.

Automated Testing: Scripts for Continuous Monitoring

For users who regularly test new models or configurations, automated benchmarking scripts save time and provide consistent measurements. A basic script: iterate over a list of model paths, run llama-bench for each, and save results to CSV. Add GPU monitoring (nvidia-smi --query-gpu=temperature.gpu,clocks.sm,power.draw --format=csv) to correlate performance drops with thermal or power throttling. For regression testing after driver updates, keep a baseline file and compare new results against it. If your TPS drops 10% after a driver update, you have objective data for a bug report. For production servers, run daily benchmarks during off-peak hours and alert if performance degrades. Tools like Prometheus with the nvidia-smi exporter can track GPU metrics over time. While this level of automation is overkill for casual users, it is valuable for anyone running inference as a daily driver where consistent performance matters.

#!/bin/bash
# Simple automated benchmark script
MODELS=("model1.gguf" "model2.gguf")
OUTPUT="benchmark_results.csv"
echo "model,gpu,backend,pp_toks,tg_toks" > "$OUTPUT"

for model in "${MODELS[@]}"; do
  result=$(./llama-bench -m "$model" -ngl 999 2>/dev/null | grep "pp\|tg")
  echo "$model,$result" >> "$OUTPUT"
done

echo "Benchmark complete. Results saved to $OUTPUT"