Instead of evolving one global harness, OO-Harness (Li et al., UIUC and Google Cloud AI Research, arXiv 2610.28380) evolves a tree of harness subclasses, each specialized for a cluster of tasks, and routes each new task to one of them after looking at a base-harness run. With the executor frozen at Gemini 3.5 Flash it lifts AutomationBench mean partial credit from 61.6% to 77.4% on held-out tasks and tops Meta-Harness, AHE and Adaptive Auto-Harness on all three benchmarks tested.
How it works
A harness here is executable code (prompts, tools, context management, control flow) wrapped around a fixed executor model. OO-Harness treats it as a class: specialized harnesses subclass the base or another specialist, override only the methods their task cluster needs, and call super() for the rest. Ancestors are not separate agent runs; the chosen class runs as one instance.
Training, on discovery tasks only (up to 40 iterations, evolution agent Claude Opus 4.8):
Run the base harness H0 on all discovery tasks and keep trajectories and grader feedback.
Cluster tasks by shared execution needs seen in those runs (not by domain). Example clusters on AutomationBench: commit-gate (tasks that end with required writes missing) and compute-verify (arithmetic or date derivations across five business domains).
Each iteration, the evolution agent picks a cluster and a parent class, states the intended change, and writes a subclass.
Run the candidate on that cluster only. Accept it if it beats the cluster's current score and passes benchmark checks (a judging agent handles ambiguous cases).
Record every outcome. Distil design lessons (what helped, what failed) for the evolution agent and routing lessons (when a harness applies) for the router. Repeated rejections lower a cluster's priority; clusters can be split, merged or reassigned.
Freeze the hierarchy, class descriptions and lessons.
Test time (router Claude Sonnet 5, no grader access):
Run H0 on the new task.
The router reads the task, that base trajectory, class descriptions and routing lessons, and either keeps the base result or picks one specialist.
If a specialist is picked, run it once and return its result. Example: repeated query-parsing errors in the base run match a routing lesson, so the router picks an API error-normalizer subclass.
The post's three-part summary (subclasses, design lessons, router) matches the paper.
Results
Held-out tasks, same Gemini 3.5 Flash executor for every method, mean of three runs (%). Primary metric: Mean for AutomationBench and Workspace-Bench, strict PASS for Toolathlon-GYM.
| Method | AutomationBench Mean | Workspace-Bench Mean | Toolathlon-GYM PASS |
|---|---|---|---|
| Base harness | 61.6 | 25.5 | 31.2 |
| Parallel best-of-5 (no grader) | 60.2 | 37.7 | 32.1 |
| Sequential self-refine, K=2 | 59.3 | 27.9 | 33.3 |
| Meta-Harness (one global harness) | 63.9 | 29.6 | 33.5 |
| AHE (Agentic Harness Engineering) | 64.2 | 38.4 | 36.2 |
| AAH (Adaptive Auto-Harness, Git branches) | 75.4 | 38.6 | 34.3 |
| OO-Harness | 77.4 | 39.5 | 37.5 |
Splits: AutomationBench 300 discovery / 300 held-out, Workspace-Bench 288 / 100 (Lite-100), Toolathlon-GYM 251 / 252.
What the numbers say:
The big win is AutomationBench, and most of it is specialization itself: AAH, the other routed-family method, gets 75.4. OO-Harness's edge over AAH is 2.0 points there, 0.9 on Workspace-Bench (inside the 3.2 standard deviation) and 3.1 on Toolathlon PASS.
Extra inference without learning does not reliably help. Best-of-5 lowers AutomationBench but helps Workspace-Bench a lot (25.5 to 37.7), nearly matching the evolved methods there.
Routing on the base trajectory matters most where clusters are about failure modes. Task-only routing gives 64.9 on AutomationBench versus 77.5 with the trajectory, but on Workspace-Bench (clusters match task types such as document writing) the gap is 0.5 points. These ablations are single runs.
Inheritance helps: on Workspace-Bench, the inheritance search scored 40.6 versus 35.6 for the same evolved code copied flat onto the base (the post's 4.96 points), and both beat resetting to base each time. Each variant is one separate search, one evaluation.
Routing sends weak cases to specialists: in one AutomationBench run, 134 of 300 tasks (44.7%) kept the base result; those had base mean reward 77.2 versus 44.4 for the 166 routed to specialists.
Out of distribution (Toolathlon-GYM training, 49 Toolathlon test tasks from disjoint app families), the router picked base for every task and kept base performance, while Meta-Harness's global harness lowered PASS.
Search was faster than Meta-Harness because candidates are scored on one cluster instead of the full discovery set.
Tradeoffs and failure modes
Every task pays for a base run. The router needs the base trajectory, so a routed task runs twice (base, then specialist) plus a router call. The reported Steps column counts only the final trajectory, so it understates cost. Compare against best-of-2, not the base, when budgeting.
The router is the new single point of silent failure. A wrong subclass pick returns a normal-looking answer. The paper reports aggregate routing ablations and a base-reuse split, but no router error rate or confusion matrix. A LinkedIn commenter asked exactly this; the paper does not answer it.
Base reuse is the safety valve. Keeping H0 as a routing option is what saved the OOD case. Any copy of this design should make "do nothing special" a first-class, cheap option.
Cluster-local acceptance can overfit small clusters. A subclass is accepted if it beats the current score on its own cluster, with no significance test. On a 10-task cluster that is noise-sensitive; R066's TRUST-style statistical gate is the obvious add.
Deep lineages are harder to read. Inheritance is a design choice for humans as much as for the search; a five-level chain of overrides with
super()calls becomes its own comprehension-debt problem. The paper does not report hierarchy depth or size.Frozen at deploy. Hierarchy, lessons and router evidence are fixed after training; there is no online update path, which is good for reproducibility and bad for drift. The limitations section only names API outages and interface changes.
Two frontier models to evolve and route a cheap one. Opus 4.8 writes the code and Sonnet 5 routes; the gains are for a Flash-class executor. The method's cost is mostly training-time and is not given in dollars.
Train/test separation is done properly. Unlike Meta-Harness's TerminalBench-2 discovery run, discovery and held-out tasks are disjoint and the router never sees grader output.
Review checklist
If you adopt specialized, routed harnesses:
Is "keep the base result" a real option, and how often is it taken in production?
Do you log every routing decision with its evidence, and sample wrong picks for review?
Is per-task cost measured including the base run and router call, not only the final trajectory?
Do specialist acceptances need more than "beat the cluster's score once" (repeats, a significance gate)?
Are clusters defined by observed failure modes where task types don't predict needs?
Is the design-lessons log kept, including rejected changes, so the next search does not repeat them?
Is there a cap on hierarchy depth, and a readable description per class?
Are discovery and evaluation tasks disjoint?
Related
R066 TRUST: a statistical acceptance gate for self-improving harnesses, the missing piece in OO-Harness's single-score acceptance.
R025 LongCapHarness: another harness-over-training result, written by hand rather than evolved.
R054 Harness engineering study: what production coding-agent harnesses actually contain, the space a search like this edits.
R103 EvoTrias: separates who chooses direction from who writes code in program evolution; OO-Harness's lessons store is a lighter version of that memory.
Sources
Elvis Saravia's LinkedIn post, about three weeks before 2026-11-16.
Li et al., Object-Oriented Harness, arXiv 2610.28380 (UIUC and Google Cloud AI Research).
DAIR.AI paper page, linked by the author.