R133 · AI-built software and its verification › Test oracles for agent-written code

VHarness: vision as a callable tool for coding agents

Read the original

verificationmultimodalcoding-agentsmcpagent-harness

Source: Yang, Wu, Jiang et al. (Peking University, Microsoft Research, University of Edinburgh, Nanjing University), "Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding", arXiv 2611.17797, November 2026; project page vision-as-a-tool.github.io. Highlighted by Pascal Biese on LinkedIn (2026-11-17).

The idea

When a coding agent turns an image into code (a webpage, an SVG, a 3D model), it has to check its rendered output against the target. Today it usually does that with its own built-in vision, sometimes with crop and zoom tools. That ties output quality to the agent's own perception, and it leaves text-only models out entirely.

Vision-as-a-Tool (VaaT) decouples the two. A separate multimodal model is exposed as a set of callable visual utilities. A text-only agent can offload all perception to it. A multimodal agent can call it when its own vision isn't enough.

The interface

VHarness exposes five atomic actions, modelled on developer tools, over MCP inside Claude Code:

There is no fixed workflow. The agent decides when to call each action, interleaving code edits, rendering and checks. Neither the baseline nor VHarness has a minimum iteration count or token budget. Both stop when the artifact is judged complete, but with VHarness that judgment comes from the external utilities rather than the agent's own eyes.

Results

Tested on Image-to-HTML, Image-to-SVG and Image-to-3D, 100 cases each, with perceptual and structural metrics (SigLIP2, DINOv2, LPIPS, SSIM, pixel MSE, edge F1, and geometric metrics for 3D):

Caveats

Why it matters beyond image-to-code

The design generalizes a pattern: when an agent's own judgment is the limit, expose a better judge as a small set of composable, typed tools (grep, inspect, assert, diff) rather than a fixed pipeline. It is the visual analogue of letting a coding agent run tests instead of reading its own code. It also lets teams pick the best coder and the best perceiver separately.

Tradeoffs

Review checklist

Connections

See R133 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.