What is a good MMLU score?

In 2026: below 50 is weak, 50-65 is average for a 7-8B model, 65-75 is strong (14-32B models), 75-85 is excellent (70B+ models, top cloud models), and 85+ is exceptional (frontier models). MMLU-Pro scores are roughly 10-15 points lower. A 7B model scoring 65 on MMLU-Pro is very strong for its size. The absolute number matters less than the relative position compared to other models in the same size class.

Which single benchmark is most important?

There is no single best benchmark. For coding: SWE-bench Verified. For knowledge breadth: MMLU-Pro. For reasoning: BBH. For math: AIME. For instruction following: IFEval. For overall user experience: Arena ELO. Different use cases prioritize different benchmarks. The closest to a 'general intelligence' benchmark is probably MMLU-Pro — it correlates reasonably well with overall capability across many tasks.

Do benchmark scores translate to real-world performance?

Partially. A model scoring 10 points higher on SWE-bench will generally be better at real-world coding — but the mapping is not 1:1. Benchmarks test specific, constrained scenarios. Real-world tasks are messier. The model that wins on benchmarks is directionally better but not guaranteed to be better for your specific workflow. Always test candidate models on your actual tasks before committing to one.

Why do some benchmarks report pass@1 and others pass@10?

Pass@1 measures the success rate on the first attempt (the model gets one try). Pass@10 measures whether at least one of ten attempts succeeds (the model gets ten tries, best result counts). Pass@10 scores are higher because the model gets more chances. For practical use, Pass@1 is more relevant — you are not going to ask the same question ten times and pick the best answer. Prefer Pass@1 scores when available.

How do I benchmark my own model?

For throughput (speed): use llama-bench from the llama.cpp project. For output quality: create a test set of 20-30 prompts covering your actual use cases. Run them through each candidate model with the same temperature and parameters. Compare outputs manually or use an LLM judge. For coding quality: use a standardized test like HumanEval through the lm-evaluation-harness framework. Keep your personal benchmark set private — if the prompts are not published online, the test is contamination-free.

LLM Benchmarks Explained: MMLU, HumanEval, SWE-bench, and More

CanItRunJuly 1, 202610 min readTutorials

Why Benchmarks Matter (and Why They Do Not)

LLM benchmarks are standardized tests designed to measure specific model capabilities: factual knowledge, reasoning ability, coding skill, mathematical aptitude, and instruction following. They provide a common yardstick for comparing models from different organizations trained on different data. Without benchmarks, choosing a model would be purely subjective — based on vibes and anecdotal experience. However, benchmarks have important limitations. They measure performance on the specific test set, which may not represent your actual use case. Models can be overfitted to benchmarks (training data accidentally including test questions). Benchmark scores do not capture important qualities like conversational tone, creativity, safety, or refusal behavior. A model with a 72 on MMLU-Pro is not necessarily 'better' than one with a 68 — it might be measurably worse at the specific tasks you care about. Use benchmarks as a filtering tool (narrow down from 20 models to 3-5 candidates), then test the finalists yourself on your actual prompts. The best model for you is the one that performs well on your tasks, not the one with the highest benchmark scores.

Knowledge Benchmarks: MMLU, MMLU-Pro, and GPQA

MMLU (Massive Multitask Language Understanding) is the most widely cited LLM benchmark. It consists of roughly 14,000 multiple-choice questions across 57 subjects — from elementary math to professional law and medicine. A score of 70 means the model answered 70% of questions correctly. MMLU measures breadth of factual knowledge rather than depth of reasoning. A high MMLU score indicates the model has memorized facts across many domains. It does not test whether the model can apply that knowledge to novel problems. MMLU-Pro (2024) is a harder variant that removes easy and ambiguous questions, focusing on 12,000+ challenging problems across 14 subjects. Scores on MMLU-Pro are roughly 10-15 points lower than MMLU for the same model. MMLU-Pro is a better discriminator between strong models. GPQA (Google-Proof Q&A, 2024) consists of 448 graduate-level questions in biology, physics, and chemistry written by domain experts. The questions are designed so that Google search does not trivially surface the answer — they require genuine expert reasoning. GPQA Diamond (the hardest 198-question subset) is particularly discriminating: scores above 50 indicate strong domain reasoning. GPQA is harder than MMLU and a better indicator of deep subject matter expertise. For choosing a model: if you need broad general knowledge, compare MMLU-Pro scores. If you need expert-level scientific reasoning, look at GPQA.

Coding Benchmarks: HumanEval, MBPP, SWE-bench, LiveCodeBench

Coding benchmarks test a model's ability to generate correct code. HumanEval (2021) is the classic: 164 hand-written Python programming problems, each with a function signature and docstring. The model must complete the function. Pass@1 measures the percentage of problems solved correctly on the first attempt. Scores above 80% are strong, above 90% are excellent. HumanEval is relatively easy by 2026 standards — most capable models score 85%+. MBPP (Mostly Basic Python Programming) is similar but with 974 problems — larger sample size, more robust scores. SWE-bench (2024) is the current gold standard for real-world coding capability. It uses 2,294 actual GitHub issues and pull requests from popular Python repositories. The model must diagnose the bug from the issue description and produce a patch that fixes it. This tests not just coding syntax but software engineering judgment: understanding codebases, diagnosing bugs, and implementing correct fixes. SWE-bench Verified (a filtered 500-problem subset removing ambiguous issues) is the most widely reported variant. A score of 50 is strong, 70+ is exceptional. Qwen 3.6 27B's 77.2 on SWE-bench Verified (tying Claude Sonnet 4.5) was a landmark for open-weight models in 2026. LiveCodeBench (2024-present) collects problems from recent coding competitions (LeetCode, AtCoder, CodeForces) to prevent benchmark contamination. Scores are generally lower and more meaningful. For choosing a coding model: SWE-bench is the best real-world indicator. HumanEval is a useful sanity check but insufficient alone — many models score 90%+ on HumanEval yet fail on practical bug-fixing tasks.

Math and Reasoning Benchmarks: MATH, AIME, BBH, MuSR

Math benchmarks test quantitative reasoning from grade school to competition level. MATH (2021) contains 12,500 problems from high school math competitions (AMC 10/12, AIME). Problems require step-by-step solutions, not just final answers. Scores have climbed rapidly: in 2022, GPT-4 scored 42.5%; in 2026, top models exceed 90%. GSM8K (2021) is grade-school math — 8,500 word problems at roughly 8th-grade level. Most capable models score 95%+, making it a poor discriminator. AIME (American Invitational Mathematics Examination) is substantially harder — problems from the actual AIME competition (held annually for top US high school math students). It is reported as pass@1 accuracy on AIME 2024 or 2025 problems. Scores above 80% (like Qwen 3.6 27B's 94.1) indicate exceptional mathematical reasoning. BBH (BIG-Bench Hard, 2022) is a suite of 23 challenging reasoning tasks testing logical deduction, causal reasoning, temporal reasoning, and more. BBH is a good general-purpose reasoning benchmark that is harder to overfit than specific math tests. MuSR (Multistep Soft Reasoning, 2024) tests a model's ability to chain multiple reasoning steps, each ambiguous in isolation but definitive when combined. High MuSR scores indicate strong narrative reasoning — the kind needed for legal analysis, investigation, and complex planning. For choosing a model for reasoning: AIME for math, BBH for general logic, MuSR for narrative/multi-step analysis.

Instruction Following and Chat: IFEval, Arena ELO, AlpacaEval

These benchmarks measure how well a model follows instructions and produces preferred outputs. IFEval (2023) tests constraint following: the prompt includes verifiable instructions like 'write at least 300 words', 'include a JSON block', or 'write entirely in uppercase'. The benchmark checks whether the output satisfies these constraints. High IFEval scores indicate a model that actually does what you ask. This is a practical metric — instruction following failure is one of the most common user frustrations. Arena ELO (Chatbot Arena, LMSYS) uses human preference: users chat with two anonymous models and vote for the better response. ELO scores are derived from pairwise comparisons, similar to chess ratings. As of mid-2026, GPT-4o and Claude Sonnet 4.5 lead the Arena with ELOs around 1300-1350. Top open-weight models reach 1250-1300. Arena ELO captures subjective user preference — it reflects tone, helpfulness, verbosity preference, and overall experience quality, not just correctness. A model with high Arena ELO but low MMLU is pleasant to chat with but factually unreliable. AlpacaEval (2023-2024) uses an LLM judge (GPT-4) to compare model outputs against a reference. It is automated (cheaper and faster than human evaluation) but inherits the judge model's biases. Length bias is a known issue: AlpacaEval tends to prefer longer responses regardless of quality. For choosing a model: Arena ELO for overall chat experience, IFEval for reliability in following instructions.

How to Use Benchmarks When Choosing a Model

A practical framework for using benchmarks: 1. Identify your primary use case. Coding? Focus on SWE-bench, LiveCodeBench, HumanEval. General knowledge? MMLU-Pro, GPQA. Math/reasoning? AIME, BBH. Creative writing? Arena ELO (subjective quality matters more than objective metrics). 2. Look at multiple benchmarks, not just one. A model scoring 85 on HumanEval but 20 on SWE-bench is good at coding puzzles but bad at real-world software engineering. Cross-reference. 3. Prioritize benchmarks that match your task difficulty. If you ask models trivia questions, MMLU matters. If you ask graduate-level science questions, GPQA matters. If you need models to write production code, SWE-bench matters. 4. Watch for overfitting. If a model scores suspiciously high on one benchmark but average on others in the same category, it may have seen the test data during training. Newer benchmarks (LiveCodeBench, GPQA) are harder to overfit. 5. Real-world testing trumps benchmarks. The best benchmark is your actual use case: run the same prompt through 3-5 candidate models and compare outputs. Benchmarks help you narrow the field. Your own evaluation makes the final decision. 6. Consider the model's size-practicality trade-off. Qwen 3.6 27B matches Claude Sonnet 4.5 on SWE-bench (77.2). It runs on a 24 GB consumer GPU. A 400B MoE model with slightly higher benchmarks that requires 8 datacenter GPUs might not be the practical choice for local inference. Use CanItRun to filter by your hardware, then compare benchmarks within your viable model set.

Benchmark Contamination: Why Scores Are Not the Full Story

Benchmark contamination occurs when a model's training data includes questions from the benchmark itself. Since most LLMs are trained on web-scale data (which includes GitHub, arXiv, and forums where benchmark questions have been discussed), contamination is likely widespread. Contamination inflates scores: the model is not reasoning through the problem but recalling the answer it saw during training. Evidence of contamination: a model scores near-perfect on a benchmark but fails on a slightly rephrased version of the same problems, or a model's score on HumanEval (old, widely published) is much higher than on LiveCodeBench (new, from recent competitions). Mitigations: newer benchmarks (LiveCodeBench, GPQA, SWE-bench Verified) use problems that were not publicly available when most models were trained. Comparing scores across old and new benchmarks can reveal contamination patterns. Community-run blind evaluations (like LMSYS Arena) avoid contamination entirely because users provide fresh prompts. For practical model selection: prefer benchmarks released after the model's training cutoff date. Weight LiveCodeBench over HumanEval. Weight GPQA over older science QA datasets. Be skeptical of unusually high scores on old, widely-published benchmarks. Trust your own testing with prompts that were never published online.

Frequently asked questions

What is a good MMLU score?: In 2026: below 50 is weak, 50-65 is average for a 7-8B model, 65-75 is strong (14-32B models), 75-85 is excellent (70B+ models, top cloud models), and 85+ is exceptional (frontier models). MMLU-Pro scores are roughly 10-15 points lower. A 7B model scoring 65 on MMLU-Pro is very strong for its size. The absolute number matters less than the relative position compared to other models in the same size class.
Which single benchmark is most important?: There is no single best benchmark. For coding: SWE-bench Verified. For knowledge breadth: MMLU-Pro. For reasoning: BBH. For math: AIME. For instruction following: IFEval. For overall user experience: Arena ELO. Different use cases prioritize different benchmarks. The closest to a 'general intelligence' benchmark is probably MMLU-Pro — it correlates reasonably well with overall capability across many tasks.
Do benchmark scores translate to real-world performance?: Partially. A model scoring 10 points higher on SWE-bench will generally be better at real-world coding — but the mapping is not 1:1. Benchmarks test specific, constrained scenarios. Real-world tasks are messier. The model that wins on benchmarks is directionally better but not guaranteed to be better for your specific workflow. Always test candidate models on your actual tasks before committing to one.
Why do some benchmarks report pass@1 and others pass@10?: Pass@1 measures the success rate on the first attempt (the model gets one try). Pass@10 measures whether at least one of ten attempts succeeds (the model gets ten tries, best result counts). Pass@10 scores are higher because the model gets more chances. For practical use, Pass@1 is more relevant — you are not going to ask the same question ten times and pick the best answer. Prefer Pass@1 scores when available.
How do I benchmark my own model?: For throughput (speed): use llama-bench from the llama.cpp project. For output quality: create a test set of 20-30 prompts covering your actual use cases. Run them through each candidate model with the same temperature and parameters. Compare outputs manually or use an LLM judge. For coding quality: use a standardized test like HumanEval through the lm-evaluation-harness framework. Keep your personal benchmark set private — if the prompts are not published online, the test is contamination-free.