R121 · Agent architecture › Self-improvement and learning loops

Harness²: self-improving harnesses without a ground-truth verifier

Read the original

self-improving-agentsagent-harnessagent-governanceevalsbenchmarks

Sources: Aishwarya Naresh Reganti's LinkedIn review (2026-10-21); Xu, Chen, CuiZhu, Dalvi Mishra, Ming, Yu, Han, Lee and Pfister (Google Cloud AI Research, Northwestern), Harness²: Recursive Agent Harnessing for an Open World, arXiv 2610.14224; code at github.com/google-research/harness2.

TL;DR

Harness² keeps the model frozen and rewrites the harness around it, per task, with no ground-truth checker. For each incoming task it runs the current harness once, proposes two edited harnesses (one aimed at general agent failures, one at domain knowledge), runs both on the same task, and lets an editor model compose a successor from the behavioral differences. After the task, it files the adopted harness plus an EXPERIENCE.md of lessons into a library that guides edits on later tasks. One step lifts GLM-5 on WorkBuddy-Bench by up to 11.5 points and on Harvey's LAB by 11.8 pp; five-way parallel recursion adds 3 to 7 more. Reganti's critique is the right frame for practitioners: the evaluator shares the executor's blind spots, and the system edits its own guardrails, so the deployable version is offline proposal, regression gate, human review, staged promotion.

What the harness is

The execution harness is eight files mounted in the runtime (extending AHE's file decomposition by one): system prompt, guardrails, memory, skills, tool guidance, scripts, subagents, and a middleware plugin. Executors are Gemini 3.5 Flash and GLM-5 under stock OpenCode and Codex. The default editor is Claude Opus 4.8 under OpenCode.

There are two harnesses in play. The execution harness does the task. The improvement harness (stage prompts, a fixed decision rule, and a growing library) decides how to edit the execution harness. The "²" is that the improver also improves, but only through its library; its prompts and rule stay fixed.

The loop

Task-level recursion (per task):

  1. Propose. Run the current harness once as a reference. The reference shows behavior, not correctness. Draft two candidates by editing component files: an invariant candidate targeting cross-domain agent failures (planning, tool use, context, delegation) and a domain candidate targeting the task's knowledge and procedures. Library entries can be reused, but task facts must be checked against the current task.

  2. Probe. Run each candidate on the same task with matched model and environment. The diff between trajectories and deliverables is a behavioral contrast: did the targeted failure go away, stay, or did something regress.

  3. Compose. An editor reads the reference, candidates, diffs and evidence and builds a successor. It may merge, revise, adopt one, or keep the current harness, and each decision must cite an observation rather than a presumed score gain. The successor is run and its output returned.

Scaling: parallel recursion draws k references and k candidates per scope, then runs k independent compositions over the full pool; sequential recursion chains k steps. For k > 1, a blind selector picks among anonymized deliverables without seeing which harness produced them.

Harnessing-level recursion (after each task): reflect on the whole improvement record; keep only lessons backed by an execution contrast ("the selector chose it" is not enough); write the adopted harness plus EXPERIENCE.md (when it applies, what was learned, the evidence) into an indexed library. The next task still starts from the base harness H0, not the last adapted one. That choice avoids drift from inheriting a harness tuned to a different task.

Results that matter

SettingBaseHarness² (k=1)+ parallel k=5
WorkBuddy avg, GLM-5 / OpenCode64.275.7+3.9
WorkBuddy avg, Gemini 3.5 Flash / OpenCode72.477.5+3.3
JobBench, Gemini / OpenCode31.536.0+3.0
LAB Partial, GLM-5 / OpenCode59.671.4+2.6

Case study: a JobBench memo task went from 43.1 to 77.8. The base run never reconciled figures across source files and never reopened its outputs. The invariant edit added a guardrail to inventory inputs, trace every figure to a source, and run a verification script on written files; the domain edit added a reconciliation skill. Both were procedural, not knowledge, fixes.

Where it breaks (Reganti's two points, sharpened)

  1. Shared blind spots. Without a verifier, "better" is model judgment over behavioral contrasts. If the editor and executor share a bias (or are the same model, which the paper shows still helps), the loop can confidently converge on the wrong behavior. The paper itself says contrasts inform composition but do not verify correctness. Benchmarks are graded externally, so the reported gains are measured with a verifier the system never sees in production.

  2. Self-edited controls. Guardrails, scripts and a middleware plugin are in the edit set. A system that can rewrite its own guardrails has rewritten its own approval. A commenter's corollary is worth keeping: in the controlled version, whoever owns the regression set owns the behavior.

Other failure modes a reviewer should raise:

Reganti's long-run point also stands: if harness improvements are this mechanical, they are candidates for distillation into the model, and today's scaffolding becomes tomorrow's default behavior.

How to run it today

Review checklist

R108 OO-Harness (same Google Cloud AI Research group; offline specialist family plus router), R066 TRUST (the statistical acceptance gate this lacks), R025 LongCapHarness, R024 InfraRSI, R040 WikiTrace (lessons tied to source runs), R069 decision models as critics, R098 CS329A self-improving agents, R026 (same post author).

Connections

See R121 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.