R108 · Agent architecture › Self-improvement and learning loops

OO-Harness: evolving a class hierarchy of agent harnesses

Read the original

self-improving-agentsagent-harnessevalsbenchmarks

Instead of evolving one global harness, OO-Harness (Li et al., UIUC and Google Cloud AI Research, arXiv 2610.28380) evolves a tree of harness subclasses, each specialized for a cluster of tasks, and routes each new task to one of them after looking at a base-harness run. With the executor frozen at Gemini 3.5 Flash it lifts AutomationBench mean partial credit from 61.6% to 77.4% on held-out tasks and tops Meta-Harness, AHE and Adaptive Auto-Harness on all three benchmarks tested.

How it works

A harness here is executable code (prompts, tools, context management, control flow) wrapped around a fixed executor model. OO-Harness treats it as a class: specialized harnesses subclass the base or another specialist, override only the methods their task cluster needs, and call super() for the rest. Ancestors are not separate agent runs; the chosen class runs as one instance.

Training, on discovery tasks only (up to 40 iterations, evolution agent Claude Opus 4.8):

  1. Run the base harness H0 on all discovery tasks and keep trajectories and grader feedback.

  2. Cluster tasks by shared execution needs seen in those runs (not by domain). Example clusters on AutomationBench: commit-gate (tasks that end with required writes missing) and compute-verify (arithmetic or date derivations across five business domains).

  3. Each iteration, the evolution agent picks a cluster and a parent class, states the intended change, and writes a subclass.

  4. Run the candidate on that cluster only. Accept it if it beats the cluster's current score and passes benchmark checks (a judging agent handles ambiguous cases).

  5. Record every outcome. Distil design lessons (what helped, what failed) for the evolution agent and routing lessons (when a harness applies) for the router. Repeated rejections lower a cluster's priority; clusters can be split, merged or reassigned.

  6. Freeze the hierarchy, class descriptions and lessons.

Test time (router Claude Sonnet 5, no grader access):

  1. Run H0 on the new task.

  2. The router reads the task, that base trajectory, class descriptions and routing lessons, and either keeps the base result or picks one specialist.

  3. If a specialist is picked, run it once and return its result. Example: repeated query-parsing errors in the base run match a routing lesson, so the router picks an API error-normalizer subclass.

The post's three-part summary (subclasses, design lessons, router) matches the paper.

Results

Held-out tasks, same Gemini 3.5 Flash executor for every method, mean of three runs (%). Primary metric: Mean for AutomationBench and Workspace-Bench, strict PASS for Toolathlon-GYM.

MethodAutomationBench MeanWorkspace-Bench MeanToolathlon-GYM PASS
Base harness61.625.531.2
Parallel best-of-5 (no grader)60.237.732.1
Sequential self-refine, K=259.327.933.3
Meta-Harness (one global harness)63.929.633.5
AHE (Agentic Harness Engineering)64.238.436.2
AAH (Adaptive Auto-Harness, Git branches)75.438.634.3
OO-Harness77.439.537.5

Splits: AutomationBench 300 discovery / 300 held-out, Workspace-Bench 288 / 100 (Lite-100), Toolathlon-GYM 251 / 252.

What the numbers say:

Tradeoffs and failure modes

Review checklist

If you adopt specialized, routed harnesses:

Sources

Connections

See R108 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.