R086 · Agent principles and patterns › Evaluation data for agents

Agents doing AI engineering: four November 2026 benchmarks

Read the original

benchmarkslong-horizonevaluationreward-hackinginference-servingrobotics

Sources: four arXiv papers from 9 to 12 November 2026: Infrathon (2611.11995, Hu, Hajishirzi, Smith, Poovendran et al., UW and Ai2), Singularity-Bench (2611.16591, Zhou, Chen, Zhang, Chen et al., CMU, NVIDIA, Together AI and others), Embodied First Exam (2611.12177, Wang, Abbeel, Song, Xing et al.) and Evolving Infra (2611.12304, Kang, Krishna, Stoica, Han et al.). Numbers below are the authors'. Surfaced together in a LinkedIn post by Tom Anusic.

The common thread

Three of these papers measure agents doing open-ended engineering over 12 to 24 hours against human-built systems, rather than solving short, checkable puzzles. The fourth asks what serving infrastructure looks like once agents are the main source of inference traffic. Together they say: long-horizon agent work is now good enough to beat mature baselines on some infrastructure tasks, the length of the eval changes the ranking, and the grading harness is itself an attack surface.

1. Infrathon: long-horizon AI infra optimization

2. Singularity-Bench: can agents build agents

3. Embodied First Exam: coding agents controlling robots

4. Evolving Infra: serving built for agent traffic

What a staff engineer should take from this

Review checklist

  1. Do our long-running agent evals report results at more than one time budget?

  2. Is the agent's "done" verified by a check it cannot edit?

  3. Is the grading environment isolated from the agent's writable workspace?

  4. Do we run more than one seed per task before ranking models?

  5. Does our serving or provider choice keep KV cache across an agent's turns?

R085 (routing model tiers on run-time signals), R042 (routing compute on checker signals), R064 (measuring skills with paired runs), R070 (generated benchmarks for data agents), R027 (provider tool-call reliability).

Connections

See R086 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.