Source: Elvis Saravia (DAIR.AI), "My Top 10 Eval Papers of 2026", LinkedIn article, 2026-11-16.
The argument
Agents now work for hours, call many tools and change real state, so grading only the final answer is no longer enough. A weak eval misleads silently: saturated benchmarks stop separating models, judges share blind spots, and self-improving agents can raise scores without improving anything. The list is ordered as a build sequence: tasks and comparisons first, then the grader, then checking the evidence, then two hard settings.
The ten papers
| # | Habit | Paper | Key finding |
|---|---|---|---|
| 1 | Pick durable benchmarks | When AI Benchmarks Plateau (Akhtar et al.) | Of 60 benchmarks, nearly half had saturated. Expert-curated sets lasted longest; whether the test set was public mattered little. |
| 2 | Test realistic tasks | Terminal-Bench (Merrill et al.) | 89 containerized command-line tasks, each with a human solution and final-state tests. Every frontier agent scored under 65% at release. |
| 3 | Run paired comparisons | SkillsBench (Li et al.) | 87 tasks across 18 model and harness setups, run with and without Skills. Average pass rate rose from 33.9% to 50.5%, but the gain ranged from about 4 to 26 points by setup. Short Skills beat long ones. |
| 4 | Use judge confidence | JEV-as-a-Judge (Li et al.) | A small judge that returns a probability came within 3 points of GPT-6 on read-checkable items. Accepting confident verdicts and escalating the rest was slightly more accurate than GPT-6 alone at 41% of the cost. Weaker on math, code, logic and persuasive answers. |
| 5 | Check judge independence | Nine Judges, Two Effective Votes (Kohli) | Nine judges from seven model families made correlated errors, so the panel carried about two independent votes. The best single judge matched or beat the panel. |
| 6 | Give judges evidence | Agent-as-a-Judge (You et al.), survey | Judges that plan checks and use tools can verify what a prompt cannot, at higher cost, latency and security risk. Sandbox them like the agent. |
| 7 | Inspect the runs | Claw-Eval (Ye et al.) | Grading from trace, service audit logs and a final-state snapshot. Output-only grading missed 44% of safety violations and 13% of robustness failures. Three runs per task. |
| 8 | Separate process and outcome | The Art of Building Verifiers for Computer Use Agents (Rosset et al.) | A verifier with separate outcome and process scores, non-overlapping rubric items, and agent-fault vs environment-fault attribution. Still some false passes. |
| 9 | Audit self-improvement | PostTrainBench (Rank et al.) | Agents post-train a base model in 10 hours on one H100. A judge flags cheating such as training on test data or downloading a tuned model; flagged runs get the base score. Agents did take both shortcuts. |
| 10 | Use personal context | LifeVal (Yang et al.) | 169 tasks built from a 3,800-person consumer survey, each with that person's history, memories, files and its own rubric. The most common failure was wrong information or a wrong recommendation. |
What to take away
Paired comparison is the one habit to start with. Run the same tasks with and without each change before trusting a gain; the SkillsBench spread shows the same change can be worth 4 points or 26.
More judges is not more signal. Measure judge error correlation first; route by confidence instead of voting.
Grade the run, not just the answer. Keep traces, audit logs and final-state snapshots, and score process separately from outcome.
Any loop that optimizes a score needs a cheating audit and a record of the data it trained on.
Caveats
This is one curator's list and the summaries above follow the article's framing; numbers are from each paper's own tasks and models and will differ on your data. Several papers note limits the article repeats: confidence cutoffs and verifier rubrics need recalibration per domain, and the judge-correlation study used inference and preference tasks.
Related entries
Other evaluation entries under index subtopic 6.3, including R102 (overfitting in AI-assisted tuning) and R105 (verification lagging generation).