Source: Together AI and Ollama, Making open models work for long-horizon agentic workloads (Together AI blog, November 2026), by Zain Hasan (Together AI) and Parth Sareen (Ollama).
Why agents are a caching problem
Every agent step resends the whole session: system prompt, tool definitions, the request, every file read and every command run. The reply is usually a short tool call. So agent turns are almost all prefill (compute-bound, sets time to first token) and very little decode (bandwidth-bound). Without reuse, redone work grows with the square of session length: with equal-sized turns, 100 turns process 5,050 units of prompt instead of 100.
Prefix caching reuses the engine's key/value state across requests, but it matches only from the start of the prompt up to the first differing token. Appending is free; changing anything early, even one character, forces everything after it to be recomputed. Everything else in the article follows from that one rule.
Harness-side rules
Keep the system prompt byte-identical. Per-turn values (timestamp, git status, current directory) at the top make every turn a full re-read. Per-session values (user name, start date) don't hurt that session but stop new sessions from reusing a shared cached setup. Put them in the task message instead.
Load tools late, at the end. Tool search that inserts a definition into the tool list near the top invalidates the whole session. Append it after the conversation instead, but some chat templates reject mid-conversation tool definitions, so support depends on the model.
Fix reasoning effort per session. On at least one open model the chat template writes effort into the first system block; raising it mid-task re-processes the entire session. Choose up front, or start fresh (or compact first) at the new level.
Truncate tool output honestly. Every tool result is paid for on every later turn, so trimming is right. Trimming silently is not: a file read that returns lines 1 to 100 of 640 with no notice leads the model to edit code it never saw. A single line saying what was cut lets it search and read the right range.
Choose subagent context deliberately. Inheriting the parent session gets a full cache hit, but the cached prefix still costs attention and GPU memory for every generated token. A reviewer needing two files and a checklist is better served by a brief that shares only the system prompt and tools. A different system prompt shares nothing.
Engine-side rules
Cache-aware routing. A session's cache lives on one node. Route by a client-supplied cache key, or by hashing the prefix to the node with the longest match (which also lets new sessions hit a shared identical setup). Load-aware routing may still choose a cold idle node over a hot overloaded one.
Keep caches alive across idle gaps. Agents wait minutes on tests. Offload idle session caches to CPU memory and restore them on the next turn rather than holding GPU memory or discarding.
Memory, not compute, is the limit. Paged allocation of cache blocks and continuous batching let bursts of subagents take slots as they free. Oversized tool results and bloated subagent contexts consume cache memory other sessions need.
Disaggregate prefill and decode. Separate clusters tuned for compute and for bandwidth, with cache moved over RDMA, so waves of long agent prompts don't slow token streaming for everyone else.
Failure modes
A dynamic value added to the system prompt for convenience silently multiplies cost and latency on long sessions.
Mid-session setting changes that look harmless (effort, tool list) are cache resets on some models.
Silent truncation produces confident edits to unseen code.
Sessions that hop nodes pay full prefill and duplicate cache storage.
Review checklist
Diff the rendered prompt between consecutive turns: does anything before the latest message change?
Is the system prompt identical across sessions of the same harness?
Where do late-loaded tool definitions land, and does the target model's chat template accept them there?
Is reasoning effort fixed per session?
Does every truncated tool result say what was omitted and how to get it?
Do subagents get a brief or the full history, and is that choice intentional?
Does your provider route by cache key or prefix, and offload idle caches?
Related entries
R113 (TokenEconomizer's tool pruning and per-repo notes), R115 (cache-affine scheduling of agent batches), and the context-economics entries under index subtopic 1.5.