Source: He, Zhoubian, Yang, Wen, Wang, Hu, Yang, Zhang, Nie, Han and Zhai, Memory Tax: Dissecting the Net Value of Agent Memory, arXiv 2610.34073 (25 Oct 2026, CC BY 4.0; project page). Surfaced by Elvis Saravia (DAIR.AI), who used it as the demo paper for a paper-to-podcast agent skill.
TL;DR
The paper asks whether dedicated agent memory systems (extract, summarize, consolidate, retrieve) are worth it compared with the simplest baseline: keep the raw conversation history and retrieve from it with plain RAG. Across several benchmarks and model backbones, many memory systems come out below that baseline, which the authors call negative net value. The proposed accounting splits a memory system's effect into a memory benefit (what organization buys you) and a memory tax (what it costs you, chiefly evidence lost or blurred when raw history is rewritten into distilled notes).
The accounting
Baseline: naive RAG over the original, unmodified history. Every evaluated system is scored relative to it, not in absolute terms.
Memory benefit: gains from structure, such as resolving references across sessions, aggregating facts, or keeping prompts short.
Memory tax: losses from the transformation itself. Summaries drop qualifiers and dates, consolidation merges facts that should stay separate, and extraction decides up front which details matter. When a later question needs a detail the memory step threw away, no retriever can recover it.
Net value = benefit minus tax. A system can be useful in absolute terms and still be negative relative to the cheap baseline.
Why it matters
Most memory systems are evaluated against no-memory or truncated-context baselines, which flatters them. Retrieval over raw history is the honest baseline and is often already available.
Memory construction costs tokens and latency at write time (summarization and consolidation calls). Third-party coverage of the paper reports cases where building memory cost more than simply passing full context, though the abstract itself gives no cost figures.
The result argues for keeping the raw log as the source of truth and treating any distilled memory as a cache or index over it, never a replacement.
Design implications
Never delete the source. Store raw turns and tool outputs; derived notes should link back to the evidence they summarize.
Retrieve evidence, not just notes. A hybrid that retrieves both the distilled note and the raw span behind it gets the structure without paying the full tax.
Measure net, not gross. Every memory feature should be A/B tested against raw-history RAG at equal token budget.
Be suspicious of lossy consolidation in long-lived agents. The tax compounds as notes are re-summarized over many sessions.
Tradeoffs and limits
Raw-history RAG grows without bound; at very long horizons, retrieval quality and storage cost become the problem memory systems were built to solve.
Some tasks really need aggregation (counting, timelines, preference drift) that chunk retrieval handles poorly; that is where memory benefit should show up.
The abstract does not name its benchmark set or cost numbers, and code is promised but not yet linked. Treat the headline as a strong prior to test on your own workload, not a settled ranking.
Review checklist
Is raw history kept, and can every memory entry be traced to its source turns?
Was the memory layer compared with RAG over raw history at the same token budget?
What does memory construction cost per session, and is it ever more than full context?
Which question types improve, and which regress, when memory is switched on?
Is consolidation ever repeated on already-summarized notes?
Related
The context-economics entries in theme 1.5 of the knowledge map, which make the same budget argument from the caching side.
R040 WikiTrace: agent memory that keeps the evidence behind each lesson, the design this paper's accounting favors.