Sources: Liu, Zhu, Wang, Li, Chen, Qiu, Qu, Lu, Xiao and Shen (Zhejiang University and Baidu), Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents, arXiv 2611.17167, 12 Nov 2026; code. Surfaced by Elvis Saravia (DAIR.AI) on LinkedIn, 2026-11-13. Numbers are the authors' own.
The question
A decision model answers with a typed output, such as a probability for each option, instead of free text. One model can therefore play three roles in an agent system:
Actor: choose the next action.
Judge: score actions that were already taken.
Teacher: score every action so a policy can be trained with reinforcement learning.
Earlier work mostly tested decision models as actors and rated them by how well they acted. This paper asks whether a model that acts badly can still judge and teach well.
What they did
Compared three decision models and three general LLMs as actors and as judges on ALFWorld (household tasks).
Used one decision model, JEV, as the teacher for a Qwen2.5-1.5B policy trained with GRPO on ALFWorld and WebShop.
Findings
All six fail the same way as actors. Each went for an easier subgoal before the one that had to come first.
The same models judge that mistake correctly. Shown the whole trajectory, they gave credit to the steps toward the subgoal they had skipped while acting.
The question matters. Asking "did this action help?" separates useful actions from useless ones. Asking "will the task succeed?" does not.
A poor actor can be a good teacher. As the GRPO teacher, JEV raised the policy from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop, including on tasks JEV rarely completes when acting.
The authors' explanation: judging after the episode ends sees the consequences that acting would have had to predict.
Why it matters for agent builders
Use small typed models as critics, not as the agent. A cheap model that outputs a distribution over "helped / didn't help" per step is a usable reward or review signal even when it can't do the task.
Give the judge the whole trajectory. Hindsight is what makes the judging work. A step-by-step judge without the outcome loses the advantage.
Ask about the step, not the task. Per-action credit questions beat outcome predictions as a training signal.
The same split applies outside RL. Reviewing a finished agent run step by step (which tool calls helped, which were wasted) is the judge role, and the paper says it can be done by a model weaker than the one that ran.
Caveats
Two text environments (ALFWorld, WebShop) and a 1.5B policy. Whether the gap between acting and judging holds for coding agents or larger policies is untested here.
Only one decision model was used as the teacher.
The paper doesn't tell you how often the judge's credit is wrong on individual steps, which matters if you use it for review rather than averaged training signal.