· Source: Ethan Mollick's LinkedIn post (2026-11-14) and the emollick/satbench repo, built on Brian Zhou's Benchmark Almanac
The claim
Mollick's post is two sentences: "AI is getting better faster than our measurements are." SatBench treats the 173 benchmarks AI labs cite most often as the questions on one exam. A question counts as solved once the best reported AI score reaches 85%. By that rule 109 of 173 (63%) are solved as of 14 Nov 2026, each year's new benchmarks fall faster than the last, and the index itself could be saturated by about 2030.
The repo is the substance. Every number comes from the almanac's own data layer, the scripts rebuild every chart, and the write-up has an unusually honest "what the data can't support" section. The headline number is much softer than the trend.
How it is built
Items. Benchmarks named in 10 or more lab launch posts, scored in percent, higher is better. Each version is its own item (AIME 2024 and AIME 2025 are two).
Solved. The best score anyone has reported reaches 85%. The score can come from an official leaderboard, Epoch AI, or the lab's own launch post.
Two views. SatBench-live adds each benchmark when it is released (of the tests that exist, how many are beaten). Fixed editions (2023, 2024, 2025) freeze the item set at year end, like an exam.
Kill. The model whose score first took an item past 85%. Born solved means a model had already passed 85% before the benchmark came out (9 items, mostly new AIME and HMMT years).
Survival analysis. Time from release to 85% uses Kaplan-Meier curves, so unsolved benchmarks count as censored rather than dropped.
What it finds
Flat, then a jump. The live index sat near 30% from 2023 to early 2025 because new tests arrived about as fast as old ones fell. It then climbed 20 points in 12 months. The 2023 edition (44 items) crossed 85% in Sep 2026, so the benchmark of benchmarks has already saturated once.
Each cohort falls faster. Share solved within a year of release: 17% (2023), 30% (2024), 43% (2025). Median time to 85% fell from about 20 months to about 14. The share of new benchmarks that are already solved at release is rising too: 0%, 2%, 7%, 14% for 2023 to 2026.
The level depends on the definition; the trend does not. Today's score is 82% at a 75% bar, 63% at 85%, 25% at 95%, and 24% counting only independent runs. Every variant is flat through 2024 and climbs in 2025-26.
Labs move to new tests. The "working set" (benchmarks in 2+ launch posts in the trailing 6 months) has stayed 41-47% solved since mid-2025. The stock is being used up, but labs swap in harder tests about as fast as they beat the old ones. In 2026 only 2 of 134 frontier launch posts named any of MMLU, HumanEval, GSM8K, MATH and the rest of the 2023 canon. Benchmarks are saturated first and dropped second.
By area. Reasoning 80% and knowledge 79% solved; multimodal 66%, code 55%, agents 45%, work 18%. Agent and work tests are most of the newest ones, and once a year old they fall as fast as any other area.
One factor. Median rank correlation between benchmarks is 0.67; after controlling for Epoch's Capabilities Index it drops to 0.06. Most of the 173 measure the same general capability. The exceptions (hallucination rate, games, GAIA, BullshitBench) are the interesting ones.
Who crosses first. Counting only the 71 benchmarks several frontier labs report: OpenAI 20, Google 14, Anthropic 14, then Qwen, ByteDance and Meta with 4 each. Anthropic has 11 of the 2026 contested kills. Raw counts mislead: Qwen names 137 of the 173 items in its own posts, so it gets credit on tests no one else runs.
Holdouts. 64 unsolved. The most cited are Humanity's Last Exam (74.2, a with-tools lab claim; the independent best is 54.8), MMMU-Pro, IFBench, Terminal-Bench 2.0, DeepSWE, AutomationBench and SciCode. 13 sit at 82 or above. Some are abandoned rather than hard, and some have a ceiling below 100% (disputed HLE answers, grader bugs, Epoch's FrontierMath ceiling of 57).
Projection, labelled an extrapolation. About 5% of unsolved items fall each month. Today's fixed set reaches 85% solved around mid-2028; the live index around 2030 if new tests keep coming and solving keeps speeding up. With no new benchmarks the last one falls around 2034. "We only run out of benchmarks if people stop making them."
Where it is weak
Three in four first crossings are a lab grading its own model. 75 of 100 kills come from the lab's launch post, and only 10 of those were later matched by an independent run. That is why the independent-only score is 24%. Launch-post scores mix harnesses, tool use and best-of-n. The data-issues list includes lab claims far above independent runs of the same model (for example Chartography, 89.0 claimed against 66.3 measured).
Admission lag. A benchmark enters only after 10 launch-post mentions, which took 2025 releases a median of about 9 months. So 2026 releases, mostly unsolved, are undercounted. With them, the live score would be about 57%, not 63%.
Thin early data. 43 launch posts from 2023-24 against about 230 a year since.
Equal weights and a fixed bar. 85% means different things on a test with label noise or a low ceiling.
The projection's key assumption is that what is left is no harder than what fell, and that new tests are not built to resist.
Commenters made the same points: the 24% against 63% gap, the admission lag, and the case for private eval sets. Mollick replied that private benchmarks "start harder, but fall just as fast."
What a staff engineer should take from it
A public benchmark has a shelf life of roughly a year. Choosing a model from a leaderboard means choosing from a mostly saturated, single-factor signal. The useful discriminators are the uncorrelated tests and your own task-level evals.
Own your eval set, and expect to replace it. If a private set falls "just as fast", the asset is not the set but the pipeline that makes new hard cases from your real work (failures, incidents, review findings). Budget for refreshing it.
Separate claimed from measured. Track who ran each number, with which harness. A vendor's launch-day score is a claim until you reproduce it.
Saturation is a trigger, not a success. When an internal eval passes 85%, raise the bar (95%, harder variants) or retire it, the way labs drop tests. Keeping a saturated eval as a gate gives false confidence.
Ceilings matter. Before chasing the last points, audit the labels; some holdouts are bugs.
Review checklist
Is the benchmark you cite more than a year old, and is it past 85% at the frontier?
Is the score independent, and is the harness (tools, best-of-n, effort level) stated?
Does the eval measure something beyond general capability (does it disagree with the leaderboard on the models you care about)?
Is there a process to add new hard cases and retire saturated ones?
Have the labels of the remaining hard cases been audited?
Related entries
R016 Mollick on BrowseComp and DeepFact: benchmarks that test finding, not checking, and planted-error sets that a constant answer can game.
R029 synthetic enterprise tenants and R070 SynthCorp: generating fresh evaluation data instead of reusing a public set, which is the answer to saturation.
R039 LLM reliability layers: measure the noise floor before claiming a gain.
R052 AACR-Bench and R064 SkillEvaluator: task-specific evals with paired runs.
R060 Gemini 4 Hydrogen and R067 Inkling 2 Small: launch posts whose vendor-reported scores are exactly the input SatBench counts.
Sources
Ethan Mollick, LinkedIn post, 2026-11-14.
Benchmark Almanac by Brian Zhou (the data source).