R045 · AI-built software and its verification › Formal methods for AI-written code

Vero: what verified repository work costs agents today

Read the original

formal-verificationai-built-softwarebenchmarksevalsinference-cost

Ye, Lou, Sun, Song, Yan, Kasriel, Zhang, He, Song, Yang and Kong (Caltech, Stanford, UC Berkeley, Apodex), arXiv 2608.13522; code at sunblaze-ucb/vero. Found via Daniel Vaughan's blog.

Vero sets the cost baseline for the strong end of formal methods: agents writing code plus Lean proofs for whole multi-module repositories. 43 Lean 4 repositories (translated from Python, Dafny, Verus and Coq sources) carry 743 scored APIs and 2,705 specs. An instance counts only if every spec is proved.

Mechanism. Two modes: code-and-proof (agent writes implementations and proofs) and proof-only (reference code given). Agents can submit machine-checked evidence that a spec is unsatisfiable or a reference is wrong, which audits the benchmark itself. Edits are limited to marked regions, axioms are allowlisted, and trivializing declarations are screened out. Harnesses were Codex and Claude Code with a 90-minute budget per run.

Results. Fully solved, code-and-proof / proof-only: GPT-5.5 xhigh 27 and 25 of 43; Claude Opus 4.8 8 and 10; GPT-5.5 medium 2 and 6; Claude Sonnet 5 2 and 2. Ten instances beat every configuration. GPT-5.5 xhigh proves 87% of individual specs, but only 51% of deep ones (helper depth 4 or more). Total cost across 43 instances: about $2,900 for GPT-5.5 xhigh, $2,000 to $2,300 for Opus 4.8, $630 to $790 for Sonnet 5.

Why it matters for efficiency. It prices the trade: roughly $67 per repository at the top setting, with no human proof review afterward. Solve rate rose much faster than spend: GPT-5.5 xhigh cost about three times medium and solved over ten times as many instances, so cheap settings are not a bargain.

Caveats. Lean only, and the corpus favors code that translates cleanly, so no concurrency or temporal protocols. No maintenance tasks (re-proving after a change). A satisfiable spec is not a correct spec, so specs were reviewed by hand. The blog's language-by-language success table and its "3 to 10 times tokens" figure are not in the paper.

Connections

See R045 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.