A coding agent can look like a request-response application during a demo: receive a task, call a model, run tools, return a patch. That model stops being useful as soon as a run outlives one process, crosses a destructive boundary or needs to recover from partial work.
The useful abstraction is a durable workflow. Each step has explicit authority, a replay strategy and evidence that survives the worker which produced it.
Why a workflow, not a loop
A loop remembers progress in memory. A workflow makes progress a durable fact. That distinction matters when a container is evicted after a migration succeeds but before the model records the result. Replaying the whole loop can apply the migration twice; resuming from a verified checkpoint can continue safely.
The control plane should therefore own transitions. Workers receive one bounded step, perform it under a lease and return a typed outcome. They do not decide that a run is durable merely because a tool call returned 0.
Define the durability boundary
The map below separates durable control state from disposable execution state. Use the Diagram and Code tabs to inspect both representations. In the diagram you can drag to pan, zoom around the pointer, use the keyboard, reset the camera or enter full screen. The source view exposes the read-only source and a copy action.
Interactive architecture map
Recoverable coding-agent workflow
The boundary is intentionally asymmetric. Sandboxes, database clones and workers may disappear. The run ledger, checkpoints and evidence bundle may not. A replacement worker reconstructs its context from those durable records instead of an old process heap or an increasingly long transcript.
A checkpoint needs enough information to prove equivalence without becoming a dump of mutable runtime state:
| Record | Durable content | Excluded content |
|---|---|---|
| Input | repository revision, step contract, capability profile | ambient shell state |
| Result | typed output, exit status, artifact hashes | raw credentials |
| Effects | idempotency keys, resource identities, compensation status | untracked side effects |
| Evidence | logs, diffs, plans, test summaries | unverifiable prose claims |
Make retries boring
Retries are safe only when the system can answer two questions: did the previous attempt commit? and would repeating it produce a second effect? The answer must come from machine-readable state, not from asking the model what it remembers.
Assign an idempotency key to every effectful boundary. A database provision request, remote comment or deployment should either return the resource created by the first attempt or refuse a conflicting replay. Local commands that cannot be made idempotent belong behind a recorded compensation step.
Classify failures before scheduling another attempt:
- A transient infrastructure failure can receive bounded exponential backoff and a new lease.
- A deterministic input failure should stop immediately with the parser, compiler or policy evidence.
- An ambiguous effect should reconcile external state before it retries.
- An irreversible or authority-expanding action should request a human decision.
This keeps the retry policy outside the model. The model proposes work; the workflow engine enforces the recovery contract.
Treat observability as state
Logs are useful for investigation, but recovery needs structured facts. Emit a transition event with the run, step, attempt, lease, revision and idempotency key. Attach tool and model traces by reference, then derive recovery SLOs from that event stream.
The most revealing measures are not token counts. Track checkpoint age, lease-expiration rate, ambiguous-effect reconciliations, compensation success and time to resume on a fresh worker. Those signals describe whether the system is actually durable.
Test the failure paths
Happy-path tests prove that the work can finish. Recovery tests prove that it can finish more than once without corruption. Inject failure immediately before and after each external effect, checkpoint write and lease renewal.
A useful minimum suite covers duplicate delivery, stale workers, lost acknowledgements, corrupted checkpoints, unavailable dependencies and human-decision timeouts. For each case, assert the final state and the absence of duplicate effects—not merely that the runner eventually returns success.
The engineering goal is deliberately unglamorous: make worker death routine. When disposable execution can vanish at any boundary and the run still resumes from verified state, long-running agents stop behaving like fragile chat sessions and start behaving like production systems.