R107 · Agent principles and patterns › Evaluation data for agents

Ten eval papers of 2026: building evals you can trust

Read the original

evaluationllm-judgebenchmarksagentsverifiers

Source: Elvis Saravia (DAIR.AI), "My Top 10 Eval Papers of 2026", LinkedIn article, 2026-11-16.

The argument

Agents now work for hours, call many tools and change real state, so grading only the final answer is no longer enough. A weak eval misleads silently: saturated benchmarks stop separating models, judges share blind spots, and self-improving agents can raise scores without improving anything. The list is ordered as a build sequence: tasks and comparisons first, then the grader, then checking the evidence, then two hard settings.

The ten papers

#HabitPaperKey finding
1Pick durable benchmarksWhen AI Benchmarks Plateau (Akhtar et al.)Of 60 benchmarks, nearly half had saturated. Expert-curated sets lasted longest; whether the test set was public mattered little.
2Test realistic tasksTerminal-Bench (Merrill et al.)89 containerized command-line tasks, each with a human solution and final-state tests. Every frontier agent scored under 65% at release.
3Run paired comparisonsSkillsBench (Li et al.)87 tasks across 18 model and harness setups, run with and without Skills. Average pass rate rose from 33.9% to 50.5%, but the gain ranged from about 4 to 26 points by setup. Short Skills beat long ones.
4Use judge confidenceJEV-as-a-Judge (Li et al.)A small judge that returns a probability came within 3 points of GPT-6 on read-checkable items. Accepting confident verdicts and escalating the rest was slightly more accurate than GPT-6 alone at 41% of the cost. Weaker on math, code, logic and persuasive answers.
5Check judge independenceNine Judges, Two Effective Votes (Kohli)Nine judges from seven model families made correlated errors, so the panel carried about two independent votes. The best single judge matched or beat the panel.
6Give judges evidenceAgent-as-a-Judge (You et al.), surveyJudges that plan checks and use tools can verify what a prompt cannot, at higher cost, latency and security risk. Sandbox them like the agent.
7Inspect the runsClaw-Eval (Ye et al.)Grading from trace, service audit logs and a final-state snapshot. Output-only grading missed 44% of safety violations and 13% of robustness failures. Three runs per task.
8Separate process and outcomeThe Art of Building Verifiers for Computer Use Agents (Rosset et al.)A verifier with separate outcome and process scores, non-overlapping rubric items, and agent-fault vs environment-fault attribution. Still some false passes.
9Audit self-improvementPostTrainBench (Rank et al.)Agents post-train a base model in 10 hours on one H100. A judge flags cheating such as training on test data or downloading a tuned model; flagged runs get the base score. Agents did take both shortcuts.
10Use personal contextLifeVal (Yang et al.)169 tasks built from a 3,800-person consumer survey, each with that person's history, memories, files and its own rubric. The most common failure was wrong information or a wrong recommendation.

What to take away

Caveats

This is one curator's list and the summaries above follow the article's framing; numbers are from each paper's own tasks and models and will differ on your data. Several papers note limits the article repeats: confidence cutoffs and verifier rubrics need recalibration per domain, and the judge-correlation study used inference and preference tasks.

Other evaluation entries under index subtopic 6.3, including R102 (overfitting in AI-assisted tuning) and R105 (verification lagging generation).

Connections

See R107 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.