Repository navigation
AgentWorkflow's cached agent stub cannot survive a mid-run DO reset — step retries all fail and onWorkflowError is silently lost #1918
Description
Summary
AgentWorkflow resolves its agent Durable Object stub once per run() execution (_initAgent → this._agent = await getAgentByName(...)) and every RPC path reuses that cached stub via the this.agent getter. If the agent DO is reset while the workflow is running — e.g. Error: Durable Object reset because its code was updated, which the platform can trigger even without a user deploy — that stub is permanently broken, and the workflow cannot recover:
- Step retries are doomed. Every retry of an in-flight step that calls
this.agent.*(includingThinkWorkflow'sstep.promptsubmit) reuses the same dead stub and fails instantly with the identical error until the retry budget is exhausted and the instance errors. - Error reporting is silently lost.
_autoReportError → notifyAgent → this.agent._workflow_handleCallbackgoes through the same broken stub, and its failure is swallowed (catch (_notifyErr) {}), so the agent'sonWorkflowErrornever fires. From the agent/UI side the workflow just vanishes.
Observed in production
agents@0.14.2, a ThinkWorkflow instance on Cloudflare Workflows. The agent DO was reset mid-run (no deploy of ours matched the window — the last deploy was ~1.5h earlier). The next step.prompt submit step failed with Durable Object reset because its code was updated., and all 6 engine retries over 5 minutes failed instantly with the byte-identical error (each attempt duration 0s — the RPC never left the broken stub). onWorkflowError never fired on the agent, so no failure ever reached our workflow-tracking DB rows or connected clients.
Name: diagnose-and-persist:submit-1
Success: ❌ No
│ 12:30:44 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:30:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:33:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:35:55 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
Still present in current main (packages/agents/src/workflows.ts): _initAgent assigns this._agent once and no RPC path re-resolves on a disconnected-stub error.
Why this seems worth fixing in the SDK
The SDK already treats this reset class as a first-class transient on the Agent side — isDurableObjectCodeUpdateReset is exported, and recent fixes (#1617, #1659, #1730) specifically recover scheduled callbacks and chat recovery from it. The workflow-side stub is the remaining gap, and it defeats one of the main reasons to use Workflows: the engine's own step retries would succeed if each attempt got a fresh stub, because the reset window is only seconds long.
Suggested fix
On an RPC failure matching isDurableObjectCodeUpdateReset (or isPlatformTransientError), re-resolve the stub with the already-captured init coordinates (getAgentByName(env[binding], name) / the facet-origin path) and retry the call once — or simply re-resolve per step attempt. submitMessages already carries an idempotency key, and the _wrapStep callbacks are idempotent, so a single retry is safe.
Workaround we're using
A subclass that overrides _initAgent at the prototype level and wraps the resolved stub in a self-healing Proxy: string-keyed method calls that reject with a code-update reset re-run the upstream _initAgent with the originally captured args and retry once; symbol-keyed members (notably Symbol.dispose, which _disposeAgent calls fire-and-forget) pass through synchronously. Happy to upstream a PR along these lines (re-resolution inside AgentWorkflow itself, using the exported classifier) if that direction sounds right.
Activity
mariusplanable commentedon Jul 28, 2026
+1 — this is a production reliability issue for us, not an edge case.
We rely on agent-attached Workflows for long-running jobs users start from the UI (often 5–30+ minutes). They kick off a task and expect it to finish or fail clearly — that’s the whole point of using Workflows.
When this bug hits, it breaks that contract:
We’ve hit this in production, including mid-run DO resets with no deploy on our side. Users see a job they started simply disappear; they often assume it succeeded or retry the same expensive work.
We shipped a workaround (self-healing stub + extra retry layers), but this belongs in the SDK. Workflows are supposed to be durable and crash-safe — a transient DO reset is exactly what step retries should absorb. A permanently cached broken stub, plus a swallowed error path, turns a seconds-long blip into a silent, unrecoverable failure.
MatthewHager commentedon Sep 24, 2026
We hit this on
agents@0.22with aThinkWorkflow, and nothing was deployed in the window. There are two things the original report doesn't cover.1. A different error message, and the classifier misses it. Our agent Durable Object sat idle for about 90 seconds while other Durable Objects did the work. The platform evicted it. The next
this.agentcall then failed with:isPlatformTransientErrorinagents@0.24.0(dist/retries.js) does not match this message. Its patterns arereset because its code was updated|this script has been upgraded,network connection lost, and the storage-reset fragment. So a fix that re-resolves the stub only onisPlatformTransientErrorwould still fail on this variant, unless the error carriesretryable: true.2. Progress calls fail the run with no retry at all.
reportProgress()→notifyAgent()→this.agent._workflow_handleCallback()is usually called fromrun()between steps, which is how the docs show it. It isn't inside astep.do, so one dead-stub call fails the whole instance. That happens even when every surrounding step retries correctly. In our case eight long child turns had all just completed (each one opens a fresh stub per attempt), and thereportProgressafter them killed the run.We're carrying a local patch along the lines of the suggested fix: re-resolve the stub with the captured init coordinates and retry once. It applies on a platform-transient error, and also on the "no longer active" disconnect. We'd happily switch to an upstream fix.