R025 · Knowledge for agents › Retrieval over long documents

LongCapHarness: a written harness instead of long-context training

Read the original

long-documentsretrievalagent-harnessself-improving-agentscontext-engineeringsmall-models

TL;DR

LongCapHarness (Zhang and He, Microsoft Research Asia, arXiv 2611.10463, 7 Nov 2026; surfaced by Elvis Saravia) replaces long-context data synthesis and post-training with a harness. A strong model (GPT-5.6-Sol) decomposes each task type into capability primitives, implements each as code or a bounded LLM call, and refines them by recursive self-improvement on a development set. The frozen harness then runs a small model with an 8K or 32K window.

How it works

A capability is an abstract input-to-output transformation shared by many questions (for example, reranking). The harnessing agent splits it into primitives with defined inputs and outputs (relevance judgement, then ordering), writes control logic that schedules them, and defines a capability state: how a new question is converted into records the primitives can use and which intermediate results to keep. Explicit rules (assignment, counting, sorting) become code; judgements become an LLM call with a bounded prompt.

The agent runs each harness on development questions, compares its answers with the references, and revises primitives, control logic and state until the gap is small. Finished harnesses are frozen into a library with a text description. At test time the small model picks a harness by description and the harness runs as a program, keeping intermediate results outside the model's context so total input length is decoupled from any one call's window.

The reranking case shows the loop's value: BM25 plus sort (40.35), then LLM scores plus sort (38.95 to 40.50), then pairwise LLM comparison with group merge (56.28 on Qwen, best at every length).

Tradeoffs and failure modes

Implications

Before post-training a small model for a long-context workload with a known set of task types, try a decomposed harness first: it is cheaper, inspectable and transfers across models for its code parts.

Sources

Connections

See R025 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.