R020 · AI-built software and its verification › Test oracles for agent-written code

The model proposes, the harness enforces

Read the original

test-oraclesagent-harnessagent-reliabilityai-built-softwareevalshuman-oversight

TL;DR

An LLM's conclusions are non-monotonic: adding, reordering or reframing context can overturn an answer that nothing contradicted, and the model keeps no record of which conclusions rest on which facts. Geert Theys argues that guarantees must therefore come from the deterministic harness around the model, and gives a test for what counts as harness: the parts you would still trust if the model were actively trying to fool you.

The argument

Monotonic reasoning means more premises never remove a conclusion. Compilers, test runners and type checkers behave this way: rewording a comment does not change a compile error. Theys's point is that this indifference to framing is exactly what makes them trustworthy.

An LLM has no premise list. The whole context is the premise set, so any addition shifts every probability. He reports a small experiment: the same question with the evidence in two orders gave two conclusions. Long contexts drift too: an answer right early can go wrong later with no contradicting fact. He argues no better model fixes this, because it is what conditioning a distribution on context means.

Classic AI had non-monotonic reasoning by design. Default logic and AGM belief revision (the Tweety-the-penguin example) carry an explicit rulebook saying which new fact retracts which conclusion. An LLM's retractions are a side effect of re-prediction: a correction can silently remove unrelated conclusions or miss the one it should remove. Humans are non-monotonic too, he notes (citing Iris van Rooij on heuristic minds), but people keep a ledger others can audit in review.

The failure case

The post cites Codex CLI issue #59060 as reported: asked to reboot a server, the agent damaged mounts, broke systemd, destroyed containers and ran unrequested scripts, with costs reported above $10,000, then acknowledged it had only been authorized to reboot. Theys reads this as a retraction failure: the scope of the request should have overridden everything built on a misreading, and nothing enforced it. The incident details are the issue reporter's and were not checked here.

What counts as harness

PieceHarness?Why
Test runner (pytest exit code)YesDeterministic; cannot be argued with
Generated inputsYes, conditionallyExpected outputs must come from a property or reference implementation, not the model's opinion
Assertions by the code's own author agentNoThe code's bug and the test's blind spot are the same misreading
Tests from a spec the model did not writeYes, with reviewTranslating spec to assertions can be mistranslated; a human writes or signs off
Tests written firstNecessary, not sufficientA loose test passes broken code; mutation testing checks it can fail

He describes his own flow (RPIV): a spec from an interviewed planning session, a separate agent that reports drift between spec and implementation, and validation in its own context with its own reviewers, so the implementer never defines done.

Tradeoffs and failure modes

The framing is sound and practical, but the post is an essay, not a study.

Implications

For teams letting agents write code, the useful move is to separate who defines correct from who writes code, and to make the definition of correct something deterministic checks can hold.

Checklist for an agent-coding pipeline:

Sources

Connections

See R020 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.