R059 · Agent architecture › Memory and context

Memory Tax: when agent memory loses to searching the raw history

Read the original

agent-memorycontext-engineeringevals

Source: He, Zhoubian, Yang, Wen, Wang, Hu, Yang, Zhang, Nie, Han and Zhai, Memory Tax: Dissecting the Net Value of Agent Memory, arXiv 2610.34073 (25 Oct 2026, CC BY 4.0; project page). Surfaced by Elvis Saravia (DAIR.AI), who used it as the demo paper for a paper-to-podcast agent skill.

TL;DR

The paper asks whether dedicated agent memory systems (extract, summarize, consolidate, retrieve) are worth it compared with the simplest baseline: keep the raw conversation history and retrieve from it with plain RAG. Across several benchmarks and model backbones, many memory systems come out below that baseline, which the authors call negative net value. The proposed accounting splits a memory system's effect into a memory benefit (what organization buys you) and a memory tax (what it costs you, chiefly evidence lost or blurred when raw history is rewritten into distilled notes).

The accounting

Why it matters

Design implications

Tradeoffs and limits

Review checklist

Connections

See R059 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.