Source: Tang, Zhang, Guo et al. with Song Han (MIT, NVIDIA, UC Berkeley, UCSD, Inco AI), Training Reasoning Models in Simulation to Natively Use Real Robots, arXiv 2611.18562, November 2026. Code: mit-han-lab/pear. All results are the authors' own.
The claim
So far, robot-use agents (a vision-language model that looks at camera images and decides where the arm goes next) have only worked well on frontier closed models such as GPT-6 Astra. Open-weight models failed even simple tasks, and the usual fix was a harness: segmentation tools, a fixed menu of motion primitives, or demonstrations for fine-tuning.
This paper trains the open-weight Qwen3.8-27B to use a robot natively, with reinforcement learning alone, entirely in simulation, with no demonstrations and no harness. The analogy is explicit: this is how computer-use agents are trained, with an arm pose in place of a click.
How one step works
camera frames + arm state + instruction + all earlier rounds
-> model reasons in a chain of thought
-> outputs next end-effector pose
-> inverse kinematics -> joint targets -> arm controller moves
-> next observation (round appended to context)
reward: 1 if the simulator says the task is done at episode end, else 0
The same loop runs in simulation and on the real robot. The model gets no perception tools and no motion menu.
Training recipe
Algorithm. GRPO-style group-normalized advantages over episodes that share a task and start state, a REINFORCE estimator without PPO clipping, and a score-centering correction for the one-to-two-version lag of fully asynchronous rollouts. Groups where every episode succeeds or every one fails are dropped (zero signal), as in DAPO.
Curriculum. With a binary reward, a task the model never solves teaches nothing. The base model succeeds in about 2% of LIBERO episodes and 0.4% of the long-horizon suite. So training starts on tasks that already give some signal, then adds tasks once transfer makes them start to succeed: three LIBERO suites, then all four, then two RoboDojo tasks.
Infrastructure (PEAR). Long multi-round episodes make prefix-cache hits essential, so the KV prefix cache is offloaded to CPU memory and concurrency is capped to keep the hit rate high. Each whole episode is one training sequence, fitted with FSDP, sequence parallelism and gradient checkpointing.
Results
| Setting | Before RL | After RURL |
|---|---|---|
| LIBERO (2,000 episodes) | 1.6% | 74.4% (near GPT-6 Astra and GPT-6.1 Sol at max effort) |
| Real dual-arm robot, never seen in training | 0 of 30 | 18 of 30 (60%) |
| LIBERO-Pro: new task in a familiar scene | — | 57.4%, vs 0.75% for the demo-trained π0.5 |
Skills transfer to a different simulator and robot, and to the held-out long-horizon suite that was excluded from the first training stage.
No measurable loss on nine reasoning, coding and vision benchmarks.
Successful real-robot trials take 7 to 24 rounds and 23k to 132k tokens of context.
Tradeoffs
Generality vs speed. No length penalty was used, so the model reasons at length before every move: about 4.3 s per round on a single GB300 at 747 tokens/s, and sometimes more than 30 s. That is fine for pick-and-place and far too slow for anything dynamic.
Harness-free vs harness. Dropping tools widens the action space (nothing is made infeasible by a motion menu), at the cost of making the model learn perception and planning itself.
RL-only vs SFT. The authors skip supervised fine-tuning to protect general capabilities, and argue robotics data belongs in pretraining or mid-training, not in SFT on a post-trained model.
Quantization. NVFP4 weights fit one consumer GPU or a laptop, but lower accuracy and lengthen reasoning, so every reported number uses BF16.
Failure modes and caveats
Small real-world sample. 30 trials on two tasks the model was trained on in simulation. The 95% interval on 18/30 is wide.
Under-trained by the authors' own account. Training stopped for lack of GPUs, after two simulators and a small task set.
Binary reward cliff. Any task the model can't solve at least sometimes contributes nothing, so the curriculum design does real work and won't transfer automatically to a new domain.
Context growth. Every round stays in context, so long tasks get slower and costlier per step.
Review checklist (for agent-RL work generally)
Does every training task give some success at the start, or is there a curriculum that makes it so?
Are all-success and all-failure groups dropped, so they don't dilute the gradient?
Is the rollout lag from asynchronous training corrected for?
Is prefix-cache hit rate measured, given that long multi-turn episodes dominate cost?
Is a held-out task family kept out of early training to test transfer honestly?
Are general benchmarks re-run after training to catch capability loss?
Is per-step latency reported, not just success rate?
Related
R126 Harness adapters: RL-tuning an open model for a specific agent harness (the opposite move: fit the model to a harness instead of removing it).
R133 VHarness: vision as a callable tool for coding agents (the harness route to giving a text model eyes).
R118 Prefix caching for coding agents (the same cache economics that PEAR's rollouts depend on).
R060 Gemini 4 Hydrogen (a frontier closed model for robotics execution).