R026 · Agent architecture › Context economics and caching

HybridAgent: agents that think in latent space and act in text

Read the original

agent-architecturereinforcement-learninginference-costagent-memoryobservabilitysmall-models

TL;DR

Hybrid Reasoning Agents (Zou, Qi, Chen, Bianchi, Zhu, Lu, He and Zou; Stanford, Together AI and UIUC; arXiv 2610.18861, code; surfaced by Aishwarya Naresh Reganti) lets a tool-using agent switch, inside one generation, between ordinary text and "latent" steps where the model feeds its own hidden state back in without decoding a token. Text is kept for tool calls, answers and the steps worth showing; the rest of the thinking happens in latent space.

How it works

HybridLink. A small residual adapter (about 2·d² parameters, roughly 10M for a 4B model) maps the last-layer hidden state back into the input embedding space. Feeding raw hidden states back, as earlier latent-reasoning methods do, scores much worse in the paper's ablation (MATH500 71.4 vs 88.2).

Mode switching. In text mode the model samples tokens as usual until it emits a <latent> token. In latent mode each step's hidden state goes through HybridLink and back into the model with no decoding. The span ends when the adapter's output is nearest to the </latent> embedding. The model decides span lengths itself.

Working memory across turns. A hybrid trace can't be re-tokenized, so the agent keeps its KV cache (text, latent steps and tool results) across tool calls and resumes from it. The paper proves this matches a full-prefix forward pass. In practice the KV cache becomes the agent's state, and the authors had to extend their serving stack, since current engines don't support it.

Training.

  1. Data. Existing text agent traces are converted to hybrid ones, with steps assigned to latent mode by LLM judges and the model's own token entropy. Tool-interaction and key steps stay in text. The SFT set is 54K trajectories.

  2. HybridSFT. Format activation teaches the anchor tokens, then the adapter is trained on frozen-backbone hidden states, then a two-pass self-conditioned stage trains the backbone on the latents its own adapter produces. That last stage is what closes the gap to plain text SFT.

  3. HybridRL. GRPO-style RL needs a token likelihood at every position, and latent steps have none. The authors show the policy factorizes over sampled text positions only, record latents during rollout and replay them exactly during the update. The reward is correctness, plus format, plus a bonus for using both Python and search. Mode usage is not rewarded; the model moves more work into latent mode as training goes on.

Evidence worth weighing

Tradeoffs and failure modes

Implications

For teams that serve their own open-weight agents, latent spans are a credible way to cut tokens and latency without giving up tool use. Everyone else should take the auditability result as the useful signal: compressing visible reasoning did not make the trace less monitorable in these tests, so trace-length budgets need not cost oversight.

Sources

Connections

See R026 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.