R035 · Agent principles and patterns › Long-running agents and durable execution

Long-horizon agents and durable execution

Read the original

durable-executionagent-harnessagent-reliabilitydistributed-systemsagent-architecturehuman-oversight

Kiran Prakash (Thoughtworks) argues on martinfowler.com (2026-11-10) that an agent which runs for weeks is a distributed system, so a durable execution runtime should absorb crashes, retries and long waits, while the harness keeps everything that needs business judgment.

The argument

The running example is a purchasing agent that keeps one SKU in stock. A replenishment takes weeks: it gathers quotes, drafts a purchase order, waits days for a manager's approval, places the order, then waits on the supplier. The agentic loop is ordinary (ask the model, run the tool it picks, repeat); what breaks it is time. A process restart, a flaky ERP call or a seven-day wait all kill an in-memory loop.

Durable execution fixes the generic part. The runtime writes each step's input and output to a log before the next step runs; a step is any call to the outside world (model call, ERP call, supplier message), and code between steps only touches local state. After a crash the program restarts from the top, recorded steps return their saved results, and the first unrecorded step runs for real. In the article's words, the runtime "keeps a record of the result of each step that interacts with the outside world" so completed steps are skipped and state is rebuilt from the record.

The second half is the useful part: durable is not the same as correct. "A persistent log records what the agent saw, not what actually happened." The runtime gives at-least-once retries, crash-safe waits and one workflow per entity. It cannot know that a timed-out call actually placed an order, that a quote went stale during a week-long wait, that the code changed under a running task, that the history has outgrown the context window, or that a supplier email is trying to steer the agent. Those stay with the harness, written as deterministic code around the model.

The prior art is old and the article names it: Write-Ahead Log, Event Sourcing, Actors and Erlang's receive ... after, Idempotent Receiver, Sagas, Optimistic Offline Lock, Parallel Change. The claim is that agents are distributed systems plus a non-deterministic component, and those patterns apply almost unchanged.

How it works

Every step is logged, so a crash replays to the last step, not the startSkuAgent: one long-lived entity workflow per SKUChecks stock, starts a replenishment, forwards manager and supplier signals to itcontinue_as_new resets its history so the log stays boundedstarts a child workflow, forwards signalsReplenish: one workflow per replenishment, finishes when doneGather quotesmodel and tools,each call a stepDraft POrules_versionpinned at startApproval waitmanager signal,price + 5% capPlace orderre-check pricePO no. as keySupplier wait7-day timeout,then chaseduplicate seen: cancel_order (saga)Highlighted: where the harness, not the runtime, keeps it correctDurable log: each step's input and output, written before the next step runsAfter a crash, replay returns recorded results and runs the first unrecorded step for real
the article's purchasing agent · entity, child workflow, durable log

One entity workflow per SKU owns a child workflow per replenishment. Waits are logged timers, so they cost nothing while idle; the place-order step is where the harness re-checks facts and supplies the idempotency key.

Failure modes: runtime vs harness

The article's table splits nine problems into four the runtime solves with general primitives and five the harness must design for. Of those five, it calls two essential (they come from the domain) and three accidental (they come from durable execution or the model).

ProblemWho handles itMechanism in the article
Process does not survive a days-long waitRuntimewait_condition with a timeout; the timer is logged, so a wait holds no process or CPU
Remote API call failsRuntimeRetryPolicy (6 attempts, 2 s initial, backoff x2) on each activity
A retry repeats workRuntime + receiverRetries are at-least-once; exactly-once effects need an idempotency key (the PO number on every place_order attempt)
A second replenishment starts while one waitsRuntimeOne long-lived entity workflow per SKU owns its child workflows; signals go to agent-SKU-123 and are forwarded
An outcome nobody sees (essential)HarnessA timed-out call is "unknown", not "failed"; a later supplier email reveals a duplicate, and a cancel_order compensation undoes it (saga)
The world changes during a wait (essential)Harnessplace_order re-fetches the quote and refuses above approved_up_to, in code, whatever the model chose
Code changes under a running task (accidental)Harnessrules_version recorded at start; new rules apply only to version 2+ (Parallel Change), old branches kept until no workflow uses them
History and memory grow without bound (accidental)Harnesscontinue_as_new resets the log; what the model sees is a curated slice (MemGPT/Letta style), not the whole history
Untrusted text steers the agent (accidental)HarnessTools carry allowed(state); only the manager's signal approves, so a supplier email saying "already approved" approves nothing

Tradeoffs and caveats

The conclusion is less technical: domain judgment decides tradeoffs such as cancelling an order vs supplier goodwill; systems of record conflict and are hard for agents to reach; and agents are only as good as their context, so data-product thinking matters. A passing mention of Thoughtworks' Agent/works platform is the only vendor note.

Review checklist for a long-running agent

  1. Is every external call (model, tool, message) a logged step, with no side effects in the code between steps?

  2. Does each side-effecting call carry an idempotency key that survives retries and replays, and does the receiver honor it?

  3. Is a timed-out or failed call treated as "unknown", with a way to discover the real outcome later?

  4. For each irreversible action, is there a compensating action and a rule for when the agent may use it?

  5. Are facts re-checked in code just before commit (price, stock, approval ceiling), not trusted from before a wait?

  6. Is there one owner per business entity, so two workflows cannot act on the same thing at once?

  7. Is every wait bounded by a timeout that becomes an event the agent can act on (chase, escalate, cancel)?

  8. Is the rules or code version recorded at workflow start, and do you know how your runtime replays old workflows on new code?

  9. Is history bounded (continue-as-new, snapshots), and is what the model sees curated separately from the durable log?

  10. Are approvals accepted only from an authenticated signal, never from text in the model's context, and are tools gated by state in code?

  11. Have you measured cold-replay latency at your expected log length?

Sources

Connections

See R035 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.