R118 · Agent architecture › Context economics and caching

Prefix caching for coding agents: what the harness and the engine must each get right

Read the original

contextcachinginferencecoding-agentsharness

Source: Together AI and Ollama, Making open models work for long-horizon agentic workloads (Together AI blog, November 2026), by Zain Hasan (Together AI) and Parth Sareen (Ollama).

Why agents are a caching problem

Every agent step resends the whole session: system prompt, tool definitions, the request, every file read and every command run. The reply is usually a short tool call. So agent turns are almost all prefill (compute-bound, sets time to first token) and very little decode (bandwidth-bound). Without reuse, redone work grows with the square of session length: with equal-sized turns, 100 turns process 5,050 units of prompt instead of 100.

Prefix caching reuses the engine's key/value state across requests, but it matches only from the start of the prompt up to the first differing token. Appending is free; changing anything early, even one character, forces everything after it to be recomputed. Everything else in the article follows from that one rule.

Harness-side rules

Engine-side rules

Failure modes

Review checklist

R113 (TokenEconomizer's tool pruning and per-repo notes), R115 (cache-affine scheduling of agent batches), and the context-economics entries under index subtopic 1.5.

Connections

See R118 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.