For the complete documentation index, see llms.txt. This page is also available as Markdown.

Learn about LLM Benchmarks

A practical guide to LLM benchmarks - covering what they measure, how scores are read, where benchmarks fall short, and which ones matter most for making smarter, evidence-based AI decisions.

Why are LLM Benchmarks Important?

1

This provides real data, not hype.

Benchmarks give you an objective way to evaluate models across reasoning, coding, math, language understanding, and safety. This helps you base decisions on measurable performance rather than claims alone.

2

Helps you match the right model to your use case.

Every model has strengths. Some lead on coding, while others perform better in reasoning or multilingual tasks. Understanding these differences helps you choose the model that best fits what you are trying to do.

3

Using multiple benchmarks gives you a clearer picture.

Models can perform very differently depending on what is being tested. Looking at multiple benchmarks gives you a fuller and more accurate view of a model’s capabilities.

4

They help guide your decision, but they do not provide definitive answers.

A strong score reflects performance under specific, controlled conditions. It does not guarantee the same results in your environment. Benchmarks should be used to narrow your options, then validated through real-world testing before making a final decision.

List of LLM Benchmarks

  • Streamlined Benchmark List: A quick reference to the benchmarks and leaderboards most commonly used to compare frontier models today.

  • Full Benchmark List: A broader reference across major capability areas, for when you need a deeper or more specific evaluation.

Benchmark
What It Evaluates
Status

MMLU-Pro

Graduate-level knowledge across multiple disciplines, using 10 answer choices instead of 4 to reduce guesswork and better separate model performance

Active - remains a strong differentiator, with scores typically lower than standard MMLU

GPQA Diamond

Expert-level scientific reasoning across biology, physics, and chemistry, built to test frontier reasoning beyond standard knowledge recall

Active differentiator - one of the strongest benchmarks for advanced scientific reasoning

AA Quality Index

A composite intelligence score from Artificial Analysis that combines results across multiple benchmarks into a single comparative metric

Active - updated as new models and benchmarks are added

Chatbot Arena (LMSYS)

Human preference in open-ended conversations, based on blind pairwise comparisons between models

Widely referenced - reflects real user preference rather than only controlled test performance

LiveCodeBench

Code generation on newly released competitive programming tasks, designed to reduce training data contamination

Active - regularly refreshed, making it one of the most current coding benchmarks available

AIME 2025

Advanced mathematical reasoning through olympiad-style problems that require multi-step problem solving

Active and highly challenging - few frontier models perform strongly here

SWE-bench Verified

Real-world software engineering through verified GitHub issue resolution across full codebases

Gold standard for coding - evaluates practical engineering ability beyond isolated code generation

Benchmark
What it Evaluates
Status

MMLU

Broad knowledge across 57 academic subjects, including STEM, humanities, and professional disciplines

Saturated

MMLU-Pro

A more difficult version of MMLU with 10 answer choices that reduce guesswork and better separate model performance

Active

GPQA Diamond

Expert-level scientific reasoning across biology, physics, and chemistry

Active

ARC-AGI 2

Abstract pattern recognition and reasoning from first principles rather than memorized knowledge

Active

Humanity's Last Exam

Extremely difficult expert-written questions across a wide range of academic domains

Active

GSM8K

Basic multi-step math word problems at grade-school level

Saturated

MATH

Competition-level mathematics that requires structured reasoning and free-form answers

Active

AIME 2025

Olympiad-level mathematical problem solving with deep multi-step reasoning

Active

HumanEval

Python function generation from natural language prompts, scored through unit-test correctness

Saturated

HumanEval+

A stricter extension of HumanEval with stronger test coverage and more edge cases

Active

LiveCodeBench

Code generation on fresh competitive programming problems updated regularly to reduce contamination risk

Active

SWE-bench Verified

Real software engineering through verified issue resolution across full codebases

Gold standard

SWE-bench Pro

Repository-level software engineering evaluation with broader language support

Emerging

IFEval

How accurately a model follows specific, verifiable instructions with constrained output requirements

Active

BFCL v4

Tool use and function calling across serial, parallel, multi-turn, and agentic workflows

Widely used

RULER

Long-context retrieval, tracking, and synthesis across extended documents

Active

MMMU Pro

Multimodal reasoning across academic subjects using both text and visual inputs

Active

TruthfulQA

Factual reliability and resistance to common misconceptions and hallucination-prone prompts

Contaminated

HELM

Multi-dimensional evaluation across accuracy, calibration, robustness, fairness, bias, and efficiency

Framework

Chatbot Arena (LMSYS)

Human preference in open-ended conversations through blind side-by-side model comparisons

Widely used

  • Active = still useful for separating model performance today

  • Saturated = top models score too closely for strong differentiation

  • Emerging = newer benchmark with growing adoption

  • Gold standard = strongest reference point in its category

  • Widely used = commonly referenced in practice

  • Contaminated = results may be less reliable due to training overlap

  • Framework = better for broad evaluation than direct ranking

Frontier Model Benchmark Snapshot (May 2026)

  • A directional comparison of leading models across publicly reported benchmarks. Blank cells indicate that a directly comparable public value was not confirmed in the source set used here.

Model
GPQA Diamond
SWE-bench Verified
ARC-AGI-2
HLE

GPT-5.4

92%

-

73.3%

39.8% no tools 52.1% with tools

GPT-5.3-Codex

83.7%

-

-

-

GPT-5.2

71.2%

72.8%

52.9%

34.5% no tools 45.5% with tools

Claude Opus 4.6

84.0%

75.6%

68.8%

40.0% no tools 53.0% with tools

Claude Sonnet 4.6

79.9%

-

58.3%

33.2% no tools 49.0% with tools

Claude Opus 4.5

86.6%

76.8%

-

-

Claude Sonnet 4.5

83.4%

71.4%

-

-

Claude Haiku 4.5

64.6%

66.6%

-

-

Gemini 3.1 Pro (Preview)

94.1%

80.6%

77.1%

44.4% no tools 51.4% with tools

Gemini 3 Flash

89.8%

75.8%

-

33.7%

Benchmarks by Use Case

  • Different tasks require different evaluation signals. This table highlights the benchmarks that are most relevant for common LLM use cases, so you can focus on the scores that best match the task at hand.

Use Case
Primary Benchmarks
Additional References

General Knowledge and Q&A

MMLU-Pro, Chatbot Arena (LMSYS)

MMLU

Code Generation

SWE-bench Verified, LiveCodeBench, SWE-bench Pro

HumanEval+, BFCL v4

Mathematical Reasoning

AIME 2025, MATH

GSM8K

Scientific Reasoning

GPQA Diamond

Humanity's Last Exam

Creative Writing

Chatbot Arena Creative Writing

-

Instruction Following

IFEval

Chatbot Arena (LMSYS)

Tool Use and Function Calling

BFCL v4

-

Long-Context Understanding

RULER, Needle-in-a-Haystack

LongGenBench

Multimodal and Vision

MMMU Pro, Arena Vision

MMMU

Multilingual Tasks

MMMLU

MLNeedle

Agentic Workflows

SWE-bench, BFCL v4

WebArena, OSWorld

Safety and Factual Reliability

HalluLens, SimpleQA

-

Last updated