Skip to content

AgentWorkflow's cached agent stub cannot survive a mid-run DO reset — step retries all fail and onWorkflowError is silently lost #1918

Open

Description

@tima-planable

Summary

AgentWorkflow resolves its agent Durable Object stub once per run() execution (_initAgent → this._agent = await getAgentByName(...)) and every RPC path reuses that cached stub via the this.agent getter. If the agent DO is reset while the workflow is running — e.g. Error: Durable Object reset because its code was updated, which the platform can trigger even without a user deploy — that stub is permanently broken, and the workflow cannot recover:

  1. Step retries are doomed. Every retry of an in-flight step that calls this.agent.* (including ThinkWorkflow's step.prompt submit) reuses the same dead stub and fails instantly with the identical error until the retry budget is exhausted and the instance errors.
  2. Error reporting is silently lost. _autoReportError → notifyAgent → this.agent._workflow_handleCallback goes through the same broken stub, and its failure is swallowed (catch (_notifyErr) {}), so the agent's onWorkflowError never fires. From the agent/UI side the workflow just vanishes.

Observed in production

agents@0.14.2, a ThinkWorkflow instance on Cloudflare Workflows. The agent DO was reset mid-run (no deploy of ours matched the window — the last deploy was ~1.5h earlier). The next step.prompt submit step failed with Durable Object reset because its code was updated., and all 6 engine retries over 5 minutes failed instantly with the byte-identical error (each attempt duration 0s — the RPC never left the broken stub). onWorkflowError never fired on the agent, so no failure ever reached our workflow-tracking DB rows or connected clients.

Name:      diagnose-and-persist:submit-1
Success:   ❌ No
│ 12:30:44 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:30:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:33:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:35:55 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │

Still present in current main (packages/agents/src/workflows.ts): _initAgent assigns this._agent once and no RPC path re-resolves on a disconnected-stub error.

Why this seems worth fixing in the SDK

The SDK already treats this reset class as a first-class transient on the Agent side — isDurableObjectCodeUpdateReset is exported, and recent fixes (#1617, #1659, #1730) specifically recover scheduled callbacks and chat recovery from it. The workflow-side stub is the remaining gap, and it defeats one of the main reasons to use Workflows: the engine's own step retries would succeed if each attempt got a fresh stub, because the reset window is only seconds long.

Suggested fix

On an RPC failure matching isDurableObjectCodeUpdateReset (or isPlatformTransientError), re-resolve the stub with the already-captured init coordinates (getAgentByName(env[binding], name) / the facet-origin path) and retry the call once — or simply re-resolve per step attempt. submitMessages already carries an idempotency key, and the _wrapStep callbacks are idempotent, so a single retry is safe.

Workaround we're using

A subclass that overrides _initAgent at the prototype level and wraps the resolved stub in a self-healing Proxy: string-keyed method calls that reject with a code-update reset re-run the upstream _initAgent with the originally captured args and retry once; symbol-keyed members (notably Symbol.dispose, which _disposeAgent calls fire-and-forget) pass through synchronously. Happy to upstream a PR along these lines (re-resolution inside AgentWorkflow itself, using the exported classifier) if that direction sounds right.

Activity

  1. mariusplanable commented on Jul 28, 2026

    @mariusplanable

    +1 — this is a production reliability issue for us, not an edge case.

    We rely on agent-attached Workflows for long-running jobs users start from the UI (often 5–30+ minutes). They kick off a task and expect it to finish or fail clearly — that’s the whole point of using Workflows.

    When this bug hits, it breaks that contract:

    1. The run appears to vanish. Error callbacks never reach the agent, so clients get no failure signal and run state isn’t updated at failure time.
    2. Engine retries don’t help. Each retry reuses the same dead stub and fails instantly with the same error, exhausting the retry budget on something that would recover in seconds with a fresh stub.
    3. Fallback reconciliation is too late. Our only recovery was a background job that eventually marked stuck runs as failed — sometimes 30 minutes later, with no user-visible signal.
      We’ve hit this in production, including mid-run DO resets with no deploy on our side. Users see a job they started simply disappear; they often assume it succeeded or retry the same expensive work.

    We shipped a workaround (self-healing stub + extra retry layers), but this belongs in the SDK. Workflows are supposed to be durable and crash-safe — a transient DO reset is exactly what step retries should absorb. A permanently cached broken stub, plus a swallowed error path, turns a seconds-long blip into a silent, unrecoverable failure.

  2. MatthewHager commented on Sep 24, 2026

    @MatthewHager

    We hit this on agents@0.22 with a ThinkWorkflow, and nothing was deployed in the window. There are two things the original report doesn't cover.

    1. A different error message, and the classifier misses it. Our agent Durable Object sat idle for about 90 seconds while other Durable Objects did the work. The platform evicted it. The next this.agent call then failed with:

    Connection closed: this Durable Object instance is no longer active. Reconnect or retry the request.
    

    isPlatformTransientError in agents@0.24.0 (dist/retries.js) does not match this message. Its patterns are reset because its code was updated|this script has been upgraded, network connection lost, and the storage-reset fragment. So a fix that re-resolves the stub only on isPlatformTransientError would still fail on this variant, unless the error carries retryable: true.

    2. Progress calls fail the run with no retry at all. reportProgress() → notifyAgent() → this.agent._workflow_handleCallback() is usually called from run() between steps, which is how the docs show it. It isn't inside a step.do, so one dead-stub call fails the whole instance. That happens even when every surrounding step retries correctly. In our case eight long child turns had all just completed (each one opens a fresh stub per attempt), and the reportProgress after them killed the run.

    We're carrying a local patch along the lines of the suggested fix: re-resolve the stub with the captured init coordinates and retry once. It applies on a platform-transient error, and also on the "no longer active" disconnect. We'd happily switch to an upstream fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions