Source: Yang, Wu, Jiang et al. (Peking University, Microsoft Research, University of Edinburgh, Nanjing University), "Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding", arXiv 2611.17797, November 2026; project page vision-as-a-tool.github.io. Highlighted by Pascal Biese on LinkedIn (2026-11-17).
The idea
When a coding agent turns an image into code (a webpage, an SVG, a 3D model), it has to check its rendered output against the target. Today it usually does that with its own built-in vision, sometimes with crop and zoom tools. That ties output quality to the agent's own perception, and it leaves text-only models out entirely.
Vision-as-a-Tool (VaaT) decouples the two. A separate multimodal model is exposed as a set of callable visual utilities. A text-only agent can offload all perception to it. A multimodal agent can call it when its own vision isn't enough.
The interface
VHarness exposes five atomic actions, modelled on developer tools, over MCP inside Claude Code:
transcribe: extract a global layout description or structural draft of the target to bootstrap code.
grep: given an image and a description, locate matching elements and return bounding boxes.
inspect: detailed observation of a region: appearance, typography, geometry.
align: a Boolean check that an image or region matches a textual specification.
diff: compare target and rendered images, and localize the mismatches.
There is no fixed workflow. The agent decides when to call each action, interleaving code edits, rendering and checks. Neither the baseline nor VHarness has a minimum iteration count or token budget. Both stop when the artifact is judged complete, but with VHarness that judgment comes from the external utilities rather than the agent's own eyes.
Results
Tested on Image-to-HTML, Image-to-SVG and Image-to-3D, 100 cases each, with perceptual and structural metrics (SigLIP2, DINOv2, LPIPS, SSIM, pixel MSE, edge F1, and geometric metrics for 3D):
VHarness improved every multimodal coding agent tested (GPT-5.5, GPT-5.6-sol, Claude Opus 4.8, GLM-5.3-Flash) across the tasks.
The largest gains went to agents with weaker native vision. On Image-to-HTML, Claude Opus 4.8's DINOv2 similarity rose from 0.72 to 0.87 and LPIPS fell from 0.46 to 0.30.
A text-only model, GLM-5.2, with a Gemini model behind the visual utilities, reached scores competitive with the native multimodal baseline GPT-5.5, and above Opus 4.8 using its own vision.
Caveats
The text-only GLM-5.2 runs used a random 50 cases per task, not 100, for budget reasons.
All targets are static: a page, a graphic or a single object. Nothing interactive or stateful was tested.
The utility model's own quality caps the result. The paradigm moves the perceptual bottleneck rather than removing it.
Each call costs a second model's inference; the paper reports quality, not cost or latency per task.
Why it matters beyond image-to-code
The design generalizes a pattern: when an agent's own judgment is the limit, expose a better judge as a small set of composable, typed tools (grep, inspect, assert, diff) rather than a fixed pipeline. It is the visual analogue of letting a coding agent run tests instead of reading its own code. It also lets teams pick the best coder and the best perceiver separately.
Tradeoffs
Modularity vs latency and cost. Every visual check is an extra model call.
External judge vs self-check. An external checker is more independent than self-assessment, but agent and judge can still share blind spots if both come from similar models.
Atomic actions vs task-specific pipelines. Five generic actions transfer across tasks; a tuned pipeline might do better on one task.
Review checklist
Does the agent verify output with something other than its own perception?
Are verification tools small, typed and composable (locate, inspect, assert, diff), or one opaque "check" call?
Is the stop decision driven by an independent check?
Have you measured the cost and latency of the extra judge calls?
Have you tested interactive or stateful targets, not just static ones?
Related
R128: render-and-self-review loops and the need for a stop rule.
R022: generator-evaluator separation with hard thresholds.
R017: fresh-context subagents as reviewers.