Source: Zou, Cai, Chen et al. (about 60 authors, GAIR-NLP), No Second Run: Does RSI Stop at Frontier Pre-training?, arXiv 2611.10719 (7 Nov 2026, CC BY 4.0). Release: GAIR-NLP/801-30B. Surfaced by DAIR.AI.
TL;DR
Recursive self-improvement (RSI) usually means many cheap loops: run, measure, change, run again. Frontier pre-training breaks that, because the full run happens once and cannot be repeated. The paper claims the first detailed account of applying RSI to an entire pre-training run from scratch. Its answer is that RSI does not stop at pre-training but changes shape: the single unrepeatable run is decomposed into many smaller loops that can each close.
What was built
801-30B-A3B-Base: a 30B-parameter mixture-of-experts model with about 3B active parameters, trained from scratch on 11.8T tokens in three months by a newly formed team, using hundreds of thousands of GPU-hours.
Post-training probe: only lightweight SFT (6B tokens, no RL), used to measure post-training headroom rather than to ship a finished chat model.
Claimed results: the base model beats Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B and trails only Qwen3.5-35B-A3B. With lightweight SFT alone it posts the highest average on four agentic benchmarks, ahead of fully post-trained models in its class. The base model and SFT data are released.
The architectural idea: close small loops inside one run
The one-shot run can't be iterated on as a whole, but its parts can. Examples are data-mixture choices tested on small proxy runs, ablations at reduced scale, checkpoint-level evaluations that feed back into the remaining schedule, and infrastructure fixes discovered mid-run. Each of these is a closed loop with its own measurement. The RSI claim is that agents and automated research can drive those inner loops, so improvement continues even though the outer loop runs once.
Tradeoffs and open questions
Proxy validity. Small-scale loops only help if their conclusions transfer to full scale. That is the classic risk of scaling-law extrapolation.
Attribution. With a new team, a new model and an automated process, it is hard to separate what RSI contributed from what ordinary good engineering would have produced. The abstract does not give an ablation without the automated loops.
Comparison fairness. Comparing a lightly fine-tuned base model with fully post-trained models on agentic benchmarks depends heavily on benchmark choice.
Cost. Hundreds of thousands of GPU-hours remains a frontier-lab budget. The method shows how to iterate within one run, not how to make the run cheap.
Review checklist
Which decisions in the run were made by automated loops, and which by people?
How was each proxy loop validated against full-scale behaviour?
Is there a counterfactual (same team and budget, no automated loops)?
Are the agentic benchmark gains reproducible from the released base model and SFT data?
Related
R024 InfraRSI: contract-locked agents for training-throughput tuning, the infrastructure-side version of closing loops inside a training run.