R052 · AI-built software and its verification › AI code review

Open Code Review: Alibaba's hybrid AI code reviewer

Read the original

code-reviewdeterministic-workflowsagent-harnessbenchmarksevalsinference-costai-built-software

Source: Mayank A.'s LinkedIn post (2026-11-12) on alibaba/open-code-review (Go, Apache-2.0, about 50k stars, v1.13.8).

TL;DR

Open Code Review (OCR) is a CLI that reviews Git changes with an LLM agent, and that Alibaba says ran as its internal reviewer for about two years. Its thesis is that a general coding agent with a review skill fails in three predictable ways: it skips files on large changes, its comments drift off the right lines, and its quality swings with small prompt edits. OCR's fix is to take the steps that must not go wrong away from the model. Plain code picks the files, bundles related files, matches rules to files, and places and re-checks comments; the agent only reads code, pulls context and reasons about defects. On Alibaba's own 200-PR benchmark this gives roughly double the F1 and four to five times the precision of Claude Code on the same model, at about a ninth of the tokens. Recall is lower, by design.

Architecture

Deterministic layer (hard constraints)

Agent layer (dynamic decisions)

Execution modes

Benchmark (AACR-Bench)

200 real PRs from 50 open-source repos in 10 languages, with 1,505 ground-truth issues labelled by 80+ senior engineers; dataset on Hugging Face (Alibaba-Aone/aacr-bench). Selected rows from the README table:

Harness + modelF1PrecisionRecallAvg timeAvg tokens
OCR v1.3.1 + Claude 4.6 Opus25.1%33.9% (301/889)20.0%1m23s385K
Claude Code v2.1.169 + Claude 4.6 Opus11.6%7.2% (435/5,980)28.9%13m6s5,664K
OCR v1.3.1 + Claude 4.8 Opus17.9%37.8% (176/465)11.7%1m6s352K
Claude Code v2.1.169 + Claude 4.8 Opus14.1%15.9% (191/1,200)12.7%5m38s2,062K
OCR v1.8.7 + Qwen3.8-Max23.0%33.9%17.4%5m14s334K
OCR v1.3.1 + GPT-5.521.0%32.1%15.5%2m51s422K
Codex v0.140.0 + GPT-5.58.4%27.8%4.9%2m58s525K

What the table actually says:

Why it matters

This is a production-scale instance of "the model proposes, the harness enforces" (R020) applied to review rather than testing: coverage, locality and rule selection are invariants, so they belong in code. It also shows that the review task decomposes cleanly into a deterministic plan (which files, which rules, where comments go) and a bounded model step, which is why delegation mode can hand the second half to any host agent.

Tradeoffs and failure modes

Review checklist (adopting an AI reviewer)

  1. Is coverage enforced by code (every changed file in some bundle), or left to the model?

  2. Are comment locations computed and validated outside the model?

  3. What is the comment volume per PR, and who triages false positives?

  4. Measure recall on your own seeded bugs, not just vendor precision.

  5. Which rules apply to which paths, and is there a catch-all pass for issues no rule names?

  6. Does a cross-bundle bug get seen by anyone?

  7. In CI, does the reviewer block merges or only comment? Low recall argues for comment-only.

  8. Is the model call delegable to your existing agent and keys, or does it need its own provider credentials?

Connections

See R052 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.