R019 · Agent architecture › Multi-agent coordination

s3: agent swarms that learn their own coordination

Read the original

multi-agentreinforcement-learningagent-architectureagent-harnessagents-at-scaleevals

TL;DR

s3 trains N copies of one small LM to organize themselves through cooperative RL: no orchestrator, no roles, no turn order, just a message/read_messages tool pair, an async harness, and one shared team reward. Source: Elvis Saravia's LinkedIn post on Jha, Faria, Li, Geng, Brahman, Smith, Tsvetkov, Kleiman-Weiner and Jaques, arXiv 2611.08615.

Setup and architecture

The whole system is a harness plus a reward; the model and prompt are generic.

Tasks and results

Two tasks pull in opposite directions as N grows, which is the paper's best design choice.

All runs train at N = 4 for 200 RL updates and evaluate at N from 1 to 16 without retraining, on 32 held-out problems × 6 seeds. Solo baselines get the same trajectories per update and per-agent token allowance.

ComparisonResultRead it as
Aggregation, N = 4s3 93% vs online SFT 35% vs individual reward 13%Strong; p < 0.001
Aggregation, N = 8s3 over 20× online SFT (the post's "25×")Controls near zero; ratio flatters
Aggregation, N = 12 / 16s3 still ahead at 12; everyone struggles at 16Exchange transfers, but doesn't scale freely
Exploration, N = 4 vs no messaging+19 points; about half the room overlapCommunication is doing real work
Exploration, N = 4 vs solo best-of-4Ahead, not significant after BonferroniNo clear win at training size
Exploration, N = 6 vs best-of-6About +7 points, no detectable token-efficiency lossThe headline "up to 9.4 points" is the best case
Exploration, large NSolo also nears 100%Benefit is reaching it with fewer agents

The headline numbers come from the abstract; the per-size readings come from section 5 of the paper.

How organization emerged

The learned protocols are simple, and they are the ones a good engineer would write by hand, which is both reassuring and a hint about how much the tasks demand.

Tradeoffs, limitations and failure modes

The method is clean; the evidence is narrow. The authors say so themselves: two tasks, small models, one training run per condition.

Implications and review checklist

You probably will not train your own swarm policy soon, but the reward and harness choices transfer directly to prompted multi-agent systems.

Checklist for a multi-agent design review:

Sources

Connections

See R019 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.