What it is
Verdict: a well-engineered orchestration pattern with honest limitation notes, not "automated research". It writes the paper; the idea and the experiments still come from people, and its prose can overstate the results it is given.
PaperOrchestra (Yiwen Song, Yale Song, Tomas Pfister, Jinsung Yoon; Google Cloud AI Research; arXiv 2604.05018) takes an idea summary, an experimental log, a LaTeX template, conference guidelines and optional figures, and returns a LaTeX manuscript and PDF. It came to wide attention through a LinkedIn post by Charly Wargnier framed as "Google just automated research papers". The repo is open source but not an official Google product, and its benchmark dataset is not yet released.
Pipeline
Five agents in five steps; plotting and literature review run in parallel, and only refinement loops.
| Agent | Output | Calls per paper | Notes |
|---|---|---|---|
| Outline | JSON plan: figure plan, literature search strategy, section plan with citation hints | 1 | Gemini-3.1-Pro, the writing backbone for all systems compared |
| Plotting | Figures and captions | 20 to 30 | Uses PaperBanana: retrieval, planning, image generation, a vision-model critic that triggers redraws |
| Literature review | Verified citation registry, .bib, drafted Introduction and Related Work | 20 to 30 | Discovery: Gemini-3-Flash with search grounding, 10 parallel workers. Verification: Semantic Scholar fuzzy title match, abstract required, must predate a cutoff date; dedup by paper id |
| Section writing | Abstract, method, experiments, conclusion; tables built from the log's numbers | 1 | May cite only verified papers; prompt requires citing at least 90% of the verified pool |
| Content refinement | Final .tex and PDF | 5 to 7 | AgentReview-style critique. A revision is kept only if the overall score rises, or ties with non-negative sub-scores; any drop reverts and halts |
The design choices worth noting: a structured plan is the only thing passed from the outline step; verification is done in code against an external index, not by asking a model; and the refinement loop is monotone (it can never accept a worse draft by its own judge's measure).
Evaluation and cost
It beats the two AI baselines clearly and still loses to human-written papers; most scores come from LLM judges that agree only moderately with people.
Benchmark. PaperWritingBench: 200 accepted papers (100 CVPR 2025, 100 ICLR 2025) reverse-engineered into idea files and experiment logs by Gemini-3.1-Pro, anonymized. Baselines: a single LLM call and AI Scientist-v2, all on the same backbone.
Automated judges. Side-by-side wins over the baselines of 88 to 99% on literature review and 39 to 88% on overall quality. Simulated acceptance by the ScholarPeer reviewer: 84% (CVPR) and 81% (ICLR) against 86% and 94% for the real papers. Citation F1 about 28 to 30 vs 17 to 28 for baselines.
Humans. 11 AI researchers, 40 papers, 180 blind paired judgments: margins of 50 to 68% on literature review and 14 to 38% on overall quality over the baselines. Against the real papers it ties or wins 43% of literature-review comparisons.
Judge agreement. GPT-5 vs humans on overall quality: Pearson 0.65. On literature review: 0.28 (GPT-5) and 0.09 (Gemini), which the authors attribute to self-preference and a taste for rigid formatting.
Ablation. Refinement wins 79 to 81% of comparisons against its own unrefined draft with no losses, and lifts simulated acceptance by 19 to 22 points.
Cost. About 60 to 70 LLM calls and 39.6 minutes per paper (AI Scientist-v2: 40 to 45 calls, 35.1 minutes; single call: 3.3 minutes). No dollar figure is given.
Failure modes
The numbers survive the pipeline; the claims around them do not always.
A commenter on the post audited released sample outputs against their source papers and reported:
Numbers faithful. 51 of 53 decimal values in one sample traced to the source; all six tables in the CVPR samples matched.
Prose not faithful. One abstract said the method "matches or exceeds" specialized models where the source said "only slightly inferior", and the sample's own table showed multi-task worse than single-task in 8 of 8 cells. A second run on the same inputs used different wording, so claim strength varies run to run.
Self-citation as prior art. The literature agent filters by date only, so the source paper (published before the cutoff) was cited in its own regenerated version, once while the text claimed to be "the first".
Quota-driven citations. Citations resolved, but relevance was not checked, and one irrelevant paper was cited; the 90%-of-pool rule pushes this way.
The paper's own limitations agree in kind: figure placement and hallucination are not controlled, there is no human in the loop, the anti-leakage instruction is a prompt that models may ignore, contamination is reduced but not ruled out, and judges reward formatting more than narrative accuracy. The single audit is one reader's check on a few samples, not a study; it is consistent with the weak judge agreement on literature quality.
The post's framing ("drops the cost of producing scientific literature to zero") overreaches: the system needs real experiment logs, and it writes from them.
Review checklist for similar pipelines
Is every comparative claim ("outperforms", "matches", "first") checked against the source numbers, not only the numbers themselves?
Does the literature step exclude the work being written up, by identity (DOI, arXiv id, repo), not just by date?
Are citation or coverage quotas absent, or paired with a relevance check?
Is the refinement judge validated against humans for each axis it scores, or only for overall quality?
Is privacy or leakage enforced in code rather than by a prompt instruction?
Are run-to-run differences measured (same inputs, several runs) before outputs are trusted?
Is there a named human gate before anything is submitted or published?
Related entries
R065 Primer: the closest pattern, narrow agents in a fixed pipeline passing inspectable artifacts, but with a human approval gate PaperOrchestra lacks.
R019 s3 swarms: the opposite design, learned coordination with no roles or orchestrator.
R031 Resolve AI: specialist agents per kind of expertise, also judged mainly by its own builders.
R016 Do humans still need to check AI's facts?: the faithfulness audit here is a counterexample to dropping human checks on claim strength.
R039 LLM reliability layers: a score-gated refinement loop sorts drafts by its judge; it does not make the judge right.
R029 Synthetic enterprise tenants: both build evaluation inputs synthetically (here, logs reverse-engineered from published papers).
R052 Open Code Review: plain code for selection and verification, models only for judgment, the same split as the citation verifier.
Sources
google-research/paper-orchestra, README
PaperOrchestra, arXiv 2604.05018: method, evaluation, limitations and cost sections
Charly Wargnier's LinkedIn post and its comment thread, including one commenter's audit