TL;DR
s3 trains N copies of one small LM to organize themselves through cooperative RL: no orchestrator, no roles, no turn order, just a message/read_messages tool pair, an async harness, and one shared team reward. Source: Elvis Saravia's LinkedIn post on Jha, Faria, Li, Geng, Brahman, Smith, Tsvetkov, Kleiman-Weiner and Jaques, arXiv 2611.08615.
Coordination task (pool private numbers): 93% success at the training size of 4 agents, vs 35% for online SFT on successful swarms and 13% for per-agent reward.
The "25×" headline is at twice the training size (8 agents) against the strongest control (online SFT), on a task where the controls had largely collapsed. It is a real generalization result, but on a toy task with Qwen3-0.6B.
Exploration task (find an exit in a tree): communicating agents revisit about half as many rooms as a no-messaging swarm, and beat best-of-6 solo by about 7 points at 6 agents. At 4 agents the edge over best-of-4 is not significant after correction.
The useful insight for builders: per-agent rewards create a public-goods dilemma where agents learn to stay silent; a max-over-team reward removes it. Hand-designed orchestration is a guess the paper argues should be learned, the bitter lesson applied to harnesses.
Setup and architecture
The whole system is a harness plus a reward; the model and prompt are generic.
Agents: N copies of one shared policy. Each has its own history, private observations, its own copy of the task tools, and a token budget L covering reasoning, tool calls and messages. Agents never see each other's reasoning or tool results.
Communication:
message(to, text)(one teammate or all) andread_messages(). Reading is optional. No message cap and no explicit messaging penalty. Names are sampled from agent_1..agent_64 per episode and shown shuffled, so no agent can learn name-specific conventions.Async scheduling: each agent resumes when its own tool call returns. Arriving messages become a notification (unread counts by sender) inserted as a tool output after the recipient's current inference request; idle agents are woken. An agent that ends a turn without a tool call goes idle and spends nothing.
Fairness on a shared inference server: generation is chopped into requests of at most 1,024 tokens so a few long generations cannot monopolize sequence slots and KV cache. The chunk boundary is also where notifications land. This is a systems detail that matters: without it one agent finishes its search before teammates can contribute.
Team reward: R = S · (1 − 0.25 · min(1, tokens spent / (N·L))), where S is 1 if any agent submits the right answer. Cost is charged only on success (charging failures rewards quitting early) and normalized by team allowance so it is size-invariant. Max, not sum (a sum rewards size) and not mean (a mean rewards each agent being right alone, which drives solve-separately-then-argue).
Credit assignment: GRPO with the team as the unit. Sample G = 8 independent teams per problem, z-normalize their rewards, and give every token of every member the team's advantage. Behavior that separates winning teams from losing ones is reinforced; per-agent credit is never computed. Loss only on tokens the agent generated, averaged per team so bigger teams do not get more weight; IcePop-style masking of off-policy tokens with ratio outside [0.2, 5].
Tasks and results
Two tasks pull in opposite directions as N grows, which is the paper's best design choice.
Information aggregation (Qwen3-0.6B, no SFT): each agent holds one private number written in English; any agent must submit all N sorted. Independent agents cannot solve it, and if each transfer succeeds with probability q, success is bounded by N·q^(N−1), so it decays exponentially unless exchange gets more reliable. Each inbox also grows to N−1 messages, eating context.
Decentralized exploration (Qwen3-1.7B, 50 SFT steps then RL): each agent explores its own copy of a depth-3, 3-way tree (391 rooms, 27 leaves, one exit) with
move/look, 8,192 tokens each. Any agent can win alone, so best-of-N already rises as 1 − (1 − p)^N; communication has to beat a baseline that improves for free.
All runs train at N = 4 for 200 RL updates and evaluate at N from 1 to 16 without retraining, on 32 held-out problems × 6 seeds. Solo baselines get the same trajectories per update and per-agent token allowance.
| Comparison | Result | Read it as |
|---|---|---|
| Aggregation, N = 4 | s3 93% vs online SFT 35% vs individual reward 13% | Strong; p < 0.001 |
| Aggregation, N = 8 | s3 over 20× online SFT (the post's "25×") | Controls near zero; ratio flatters |
| Aggregation, N = 12 / 16 | s3 still ahead at 12; everyone struggles at 16 | Exchange transfers, but doesn't scale freely |
| Exploration, N = 4 vs no messaging | +19 points; about half the room overlap | Communication is doing real work |
| Exploration, N = 4 vs solo best-of-4 | Ahead, not significant after Bonferroni | No clear win at training size |
| Exploration, N = 6 vs best-of-6 | About +7 points, no detectable token-efficiency loss | The headline "up to 9.4 points" is the best case |
| Exploration, large N | Solo also nears 100% | Benefit is reaching it with fewer agents |
The headline numbers come from the abstract; the per-size readings come from section 5 of the paper.
How organization emerged
The learned protocols are simple, and they are the ones a good engineer would write by hand, which is both reassuring and a hint about how much the tasks demand.
Aggregation converges to "broadcast once, early." Early in training about half of swarms have a silent agent and most of the rest chatter by direct message. By the end about 80% use exactly one broadcast per agent, and almost no agent reads its inbox before sending. Swarms that happened to broadcast once early on succeeded about 30 points more often, so the team reward selected for it.
Per-agent reward produces a public-goods dilemma. If you are rewarded only for your own correct list, your number only helps others, so the best move is to read first and wait. Agents learn exactly that and many never send. The shared max reward removes the incentive. In exploration there is no dilemma (a teammate's report helps you win yourself), so individual reward does about as well as shared reward there.
Explorers split the tree. Every agent sees doors in the same fixed order, so agents with the same habits pile down door 1. Without messages, success drops sharply when the exit is behind door 3. s3 swarms succeed at about the same rate wherever the exit is, because agents claim branches and report finished ones.
Imitation is not enough. Online SFT on the swarm's own successful episodes got 35% vs 93%: copying winning traces does not teach when a message matters, which the RL signal does.
Tradeoffs, limitations and failure modes
The method is clean; the evidence is narrow. The authors say so themselves: two tasks, small models, one training run per condition.
Toy tasks with checkable answers. Sorting N numbers and maze search have binary, verifiable success. Real multi-agent work (code changes, research) has partial credit, noisy verifiers and long horizons, where a team-level max reward gives very sparse signal.
Small models, small teams. 0.6B and 1.7B params, trained at N = 4. Whether learned protocols hold with frontier models that already follow prompted collaboration well is untested; the gap over a good prompt could shrink.
Exploration win is modest. Not significant at the training size after correction; the best case is +9.4 points; at large N best-of-N catches up. The practical claim is "same success with fewer agents."
Exponential wall in aggregation. All methods struggle at 16 agents. Broadcast is N(N−1) deliveries and each inbox eats context; learning makes exchange more reliable but does not change the scaling law.
Trust failures remain. Agents still submit wrong answers with all the data in hand, repeat a teammate's search, or trust an unverified report. No mechanism checks a teammate's claim, which matters once reports can be wrong or adversarial.
Training needs the harness. The gains depend on async scheduling and 1,024-token chunking on the inference server. A model trained this way and then dropped into a synchronous, turn-based orchestrator would likely lose some of the benefit (my inference; the paper does not test it).
Implications and review checklist
You probably will not train your own swarm policy soon, but the reward and harness choices transfer directly to prompted multi-agent systems.
Score the team, not the agent. Any per-agent metric (each subagent graded on its own output) recreates the read-first, never-share dilemma. Evaluate on the final team outcome.
Use best-of-N at matched budget as the baseline. Many multi-agent wins disappear against N independent attempts with the same tokens. If your orchestrator cannot beat that, it is overhead.
Prefer broadcast-once for pooling facts, claims for search. Those are the two protocols that emerged; they are cheap to prompt for.
Keep messages separate from private reasoning. s3 agents share only what they choose to send, which bounds context growth and keeps each agent's working memory its own.
Make the harness async and preemptible. Slow agents should not block fast ones, and long generations should yield so peers' findings arrive while still useful.
Checklist for a multi-agent design review:
Is there a matched-budget best-of-N baseline, and does the system beat it?
Is success scored at the team level, with no per-agent incentive to withhold?
Does the communication cost grow as N² (broadcast) and is the context budget sized for it?
Can one slow agent stall the rest?
Does any agent act on a teammate's report without checking it?
Has the design been tested at a team size other than the one it was tuned on?
Related entries
R015 (workflows over agents): argues extra agents are only worth it for parallel work and that most orchestration is a workflow in disguise. s3's exploration result agrees on parallelism, and its best-of-N baseline is the test it implies.
R006 (harness building blocks): s3's harness is a minimal instance: an agent loop, two messaging tools, async tool execution and a token budget as the only guardrail.
R017 and R016 (verification): s3's remaining failure, trusting an unverified teammate report, is what independent zero-context reviewers and verifier routing are meant to catch.