R069 · Agent architecture › Self-improvement and learning loops

Actor, judge or teacher: decision models are better critics than agents

Read the original

self-improving-agentsdecision-modelsevalsinference-cost

Sources: Liu, Zhu, Wang, Li, Chen, Qiu, Qu, Lu, Xiao and Shen (Zhejiang University and Baidu), Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents, arXiv 2611.17167, 12 Nov 2026; code. Surfaced by Elvis Saravia (DAIR.AI) on LinkedIn, 2026-11-13. Numbers are the authors' own.

The question

A decision model answers with a typed output, such as a probability for each option, instead of free text. One model can therefore play three roles in an agent system:

Earlier work mostly tested decision models as actors and rated them by how well they acted. This paper asks whether a model that acts badly can still judge and teach well.

What they did

Findings

  1. All six fail the same way as actors. Each went for an easier subgoal before the one that had to come first.

  2. The same models judge that mistake correctly. Shown the whole trajectory, they gave credit to the steps toward the subgoal they had skipped while acting.

  3. The question matters. Asking "did this action help?" separates useful actions from useless ones. Asking "will the task succeed?" does not.

  4. A poor actor can be a good teacher. As the GRPO teacher, JEV raised the policy from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop, including on tasks JEV rarely completes when acting.

The authors' explanation: judging after the episode ends sees the consequences that acting would have had to predict.

Why it matters for agent builders

Caveats

Connections

See R069 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.