R138 · Agent architecture › Self-improvement and learning loops

RURL: training an open model to use a real robot the way agents use a computer

Read the original

reinforcement-learningcomputer-useopen-modelsroboticsharnessbenchmarks

Source: Tang, Zhang, Guo et al. with Song Han (MIT, NVIDIA, UC Berkeley, UCSD, Inco AI), Training Reasoning Models in Simulation to Natively Use Real Robots, arXiv 2611.18562, November 2026. Code: mit-han-lab/pear. All results are the authors' own.

The claim

So far, robot-use agents (a vision-language model that looks at camera images and decides where the arm goes next) have only worked well on frontier closed models such as GPT-6 Astra. Open-weight models failed even simple tasks, and the usual fix was a harness: segmentation tools, a fixed menu of motion primitives, or demonstrations for fine-tuning.

This paper trains the open-weight Qwen3.8-27B to use a robot natively, with reinforcement learning alone, entirely in simulation, with no demonstrations and no harness. The analogy is explicit: this is how computer-use agents are trained, with an arm pose in place of a click.

How one step works

camera frames + arm state + instruction + all earlier rounds
  -> model reasons in a chain of thought
  -> outputs next end-effector pose
  -> inverse kinematics -> joint targets -> arm controller moves
  -> next observation (round appended to context)
reward: 1 if the simulator says the task is done at episode end, else 0

The same loop runs in simulation and on the real robot. The model gets no perception tools and no motion menu.

Training recipe

Results

SettingBefore RLAfter RURL
LIBERO (2,000 episodes)1.6%74.4% (near GPT-6 Astra and GPT-6.1 Sol at max effort)
Real dual-arm robot, never seen in training0 of 3018 of 30 (60%)
LIBERO-Pro: new task in a familiar scene—57.4%, vs 0.75% for the demo-trained π0.5

Tradeoffs

Failure modes and caveats

Review checklist (for agent-RL work generally)

Connections

See R138 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.