R077 · Agent architecture › Multi-agent coordination

PaperOrchestra: a five-agent pipeline that writes research papers from logs

Read the original

multi-agentresearch-automationagent-workflowsevalsfact-checkinghuman-oversight

What it is

Verdict: a well-engineered orchestration pattern with honest limitation notes, not "automated research". It writes the paper; the idea and the experiments still come from people, and its prose can overstate the results it is given.

PaperOrchestra (Yiwen Song, Yale Song, Tomas Pfister, Jinsung Yoon; Google Cloud AI Research; arXiv 2604.05018) takes an idea summary, an experimental log, a LaTeX template, conference guidelines and optional figures, and returns a LaTeX manuscript and PDF. It came to wide attention through a LinkedIn post by Charly Wargnier framed as "Google just automated research papers". The repo is open source but not an official Google product, and its benchmark dataset is not yet released.

Pipeline

Five agents in five steps; plotting and literature review run in parallel, and only refinement loops.

score rose: keep, iteratescore fell or limit: revert, stop

Idea + experiment log + template

1 Outline agent

2 Plotting agent

3 Literature review agent

4 Section writing agent

5 Content refinement

LaTeX + PDF

AgentOutputCalls per paperNotes
OutlineJSON plan: figure plan, literature search strategy, section plan with citation hints1Gemini-3.1-Pro, the writing backbone for all systems compared
PlottingFigures and captions20 to 30Uses PaperBanana: retrieval, planning, image generation, a vision-model critic that triggers redraws
Literature reviewVerified citation registry, .bib, drafted Introduction and Related Work20 to 30Discovery: Gemini-3-Flash with search grounding, 10 parallel workers. Verification: Semantic Scholar fuzzy title match, abstract required, must predate a cutoff date; dedup by paper id
Section writingAbstract, method, experiments, conclusion; tables built from the log's numbers1May cite only verified papers; prompt requires citing at least 90% of the verified pool
Content refinementFinal .tex and PDF5 to 7AgentReview-style critique. A revision is kept only if the overall score rises, or ties with non-negative sub-scores; any drop reverts and halts

The design choices worth noting: a structured plan is the only thing passed from the outline step; verification is done in code against an external index, not by asking a model; and the refinement loop is monotone (it can never accept a worse draft by its own judge's measure).

Evaluation and cost

It beats the two AI baselines clearly and still loses to human-written papers; most scores come from LLM judges that agree only moderately with people.

Failure modes

The numbers survive the pipeline; the claims around them do not always.

A commenter on the post audited released sample outputs against their source papers and reported:

The paper's own limitations agree in kind: figure placement and hallucination are not controlled, there is no human in the loop, the anti-leakage instruction is a prompt that models may ignore, contamination is reduced but not ruled out, and judges reward formatting more than narrative accuracy. The single audit is one reader's check on a few samples, not a study; it is consistent with the weak judge agreement on literature quality.

The post's framing ("drops the cost of producing scientific literature to zero") overreaches: the system needs real experiment logs, and it writes from them.

Review checklist for similar pipelines

Sources

Connections

See R077 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.