R078 · Agent principles and patterns › Evaluation data for agents

SatBench: how fast AI benchmarks saturate

Read the original

benchmarksevalsbenchmark-saturationmodel-releasesai-impact

· Source: Ethan Mollick's LinkedIn post (2026-11-14) and the emollick/satbench repo, built on Brian Zhou's Benchmark Almanac

The claim

Mollick's post is two sentences: "AI is getting better faster than our measurements are." SatBench treats the 173 benchmarks AI labs cite most often as the questions on one exam. A question counts as solved once the best reported AI score reaches 85%. By that rule 109 of 173 (63%) are solved as of 14 Nov 2026, each year's new benchmarks fall faster than the last, and the index itself could be saturated by about 2030.

The repo is the substance. Every number comes from the almanac's own data layer, the scripts rebuild every chart, and the write-up has an unusually honest "what the data can't support" section. The headline number is much softer than the trend.

How it is built

What it finds

  1. Flat, then a jump. The live index sat near 30% from 2023 to early 2025 because new tests arrived about as fast as old ones fell. It then climbed 20 points in 12 months. The 2023 edition (44 items) crossed 85% in Sep 2026, so the benchmark of benchmarks has already saturated once.

  2. Each cohort falls faster. Share solved within a year of release: 17% (2023), 30% (2024), 43% (2025). Median time to 85% fell from about 20 months to about 14. The share of new benchmarks that are already solved at release is rising too: 0%, 2%, 7%, 14% for 2023 to 2026.

  3. The level depends on the definition; the trend does not. Today's score is 82% at a 75% bar, 63% at 85%, 25% at 95%, and 24% counting only independent runs. Every variant is flat through 2024 and climbs in 2025-26.

  4. Labs move to new tests. The "working set" (benchmarks in 2+ launch posts in the trailing 6 months) has stayed 41-47% solved since mid-2025. The stock is being used up, but labs swap in harder tests about as fast as they beat the old ones. In 2026 only 2 of 134 frontier launch posts named any of MMLU, HumanEval, GSM8K, MATH and the rest of the 2023 canon. Benchmarks are saturated first and dropped second.

  5. By area. Reasoning 80% and knowledge 79% solved; multimodal 66%, code 55%, agents 45%, work 18%. Agent and work tests are most of the newest ones, and once a year old they fall as fast as any other area.

  6. One factor. Median rank correlation between benchmarks is 0.67; after controlling for Epoch's Capabilities Index it drops to 0.06. Most of the 173 measure the same general capability. The exceptions (hallucination rate, games, GAIA, BullshitBench) are the interesting ones.

  7. Who crosses first. Counting only the 71 benchmarks several frontier labs report: OpenAI 20, Google 14, Anthropic 14, then Qwen, ByteDance and Meta with 4 each. Anthropic has 11 of the 2026 contested kills. Raw counts mislead: Qwen names 137 of the 173 items in its own posts, so it gets credit on tests no one else runs.

  8. Holdouts. 64 unsolved. The most cited are Humanity's Last Exam (74.2, a with-tools lab claim; the independent best is 54.8), MMMU-Pro, IFBench, Terminal-Bench 2.0, DeepSWE, AutomationBench and SciCode. 13 sit at 82 or above. Some are abandoned rather than hard, and some have a ceiling below 100% (disputed HLE answers, grader bugs, Epoch's FrontierMath ceiling of 57).

Projection, labelled an extrapolation. About 5% of unsolved items fall each month. Today's fixed set reaches 85% solved around mid-2028; the live index around 2030 if new tests keep coming and solving keeps speeding up. With no new benchmarks the last one falls around 2034. "We only run out of benchmarks if people stop making them."

Where it is weak

Commenters made the same points: the 24% against 63% gap, the admission lag, and the case for private eval sets. Mollick replied that private benchmarks "start harder, but fall just as fast."

What a staff engineer should take from it

Review checklist

Sources

Connections

See R078 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.