fix(agents,think): recover server-tool turns, terminalize thrown recovery, re-run alarms killed by a script-upgrade supersede

[threepointone]

Summary

Three chat-recovery / durable-execution failures that could leave a turn wedged at a half-finished assistant message after a deploy/eviction, with no terminal banner — the turn never recovers and never terminalizes until something unrelated wakes the Durable Object. All three were reproduced from production incidents on a server-tool-heavy app (a preview tool whose execute() bridges to the SPA over HTTP and parks awaiting external I/O).

1. Server-tool recovery deadlock — recovery can never converge for a server tool interrupted mid-execute(). (think)
2. Silent seal on a thrown recovery callback — a non-reset throw in the recovery continuation is swallowed and the alarm row deleted, so no onExhausted / banner ever fires. (think)
3. One-shot alarm killed by a "script upgraded" supersede is not re-run — only "reset because its code was updated" was recognized as a deferrable platform kill, so the sibling supersede message orphaned the work with no driver. (agents core + think)

This is a follow-up in the same #1649 / #1651 / #1657 recovery-hardening line; it targets the server-tool + deploy-recovery half of the "interrupted tool" symptom (whereas #1657 fixed the client-tool + normal/auto-continuation half). See "Relationship to #1657" below.

---

Bug 1 — server-tool recovery deadlock

Repro

1. App registers a server-side tool whose execute() awaits external I/O (e.g. a long-poll bridge: the SPA fetches the tool input and posts the result back over HTTP; the in-flight promise lives only in DO memory).
2. Model calls the tool; the tool-input-available chunk is flushed durably.
3. Worker is redeployed while execute() is parked. The DO is evicted; the in-memory promise is gone.
4. On restart, _handleInternalFiberRecovery persists the orphaned partial (the resumable stream is restored from SQLite, so streamStillActive is true → _persistOrphanedStream runs), landing a new assistant message with the tool part at state: "input-available".
5. Recovery schedules _chatRecoveryContinue, which calls waitUntilStable before continueLastTurn. hasPendingInteraction() returns true because of the input-available part — but nothing will ever resolve it: there is no client tool-result for a server tool, no in-memory _pendingInteractionPromise (it died with the isolate), and _repairToolTranscriptParts (which would flip it to an errored result) only runs inside _runInferenceLoop, i.e. downstream of the very gate that won't converge.
6. waitUntilStable times out every attempt; the continuation reschedules until the budget drains (pure waste — the wait can never converge).

hasPendingInteraction can't tell "a client is about to post a result over WS" from "a server tool's execute() promise died with the DO". For a server tool the second case is permanent.

Fix

Narrow the recovery stability predicate. An input-available part is now only "pending" when it is genuinely client-resolvable:

• approval-requested — a reconnecting client can replay the approval; or
• input-available for a client tool (its name is in the last request's client-tool set) — the SPA can replay the tool-result.

A server tool's input-available is no longer treated as pending. waitUntilStable converges, continueLastTurn runs, and the existing transcript-repair pass flips the orphan to output-error so the model continues with an errored tool result.

Why client-tool recovery is preserved: client tools are persisted to config at request acceptance (_persistClientTools, before any tool streams) and restored in onStart (_restoreClientTools), which alarm() awaits before dispatching the recovery callback. So _lastClientTools is always populated during the wait and a client-tool orphan is correctly kept "pending". waitUntilStable / hasPendingInteraction are recovery-only (their sole callers are _chatRecoveryContinue / _chatRecoveryRetry), so the change is tightly scoped and can't affect a normal turn.

---

Bug 2 — silent seal on a thrown recovery callback

Root cause

onExhausted only fired on the clean budget-exhaustion branch. Any error thrown by _chatRecoveryContinue / _chatRecoveryRetry after the incident was opened was marked failed and re-thrown. Agent._executeScheduleCallback re-throws only a Durable Object code-update reset (to preserve the one-shot row for re-run on the fresh isolate); it swallows any other error and returns normally, after which alarm() deletes the one-shot row. Net: the turn is sealed with no onExhausted event and no terminal banner — the half-finished message wedges silently until the user sends something new. (This matched the production trace: "recovery alarm threw, no terminal event, message frozen ~6 min until the user typed again".)

Fix

Route a non-reset throw through the same exhaustion path as a budget give-up (_exhaustRecoveryGiveUp, reason recovery_error) so onExhausted fires and the configured terminalMessage banner is delivered, then return without re-throwing. A genuine code-update / supersede reset is still re-thrown so the platform re-runs recovery on the fresh isolate (see Bug 3). The underlying error is logged for operators (the give-up records only the recovery_error category).

Hardening (closes the original telemetry observation)

• _exhaustChatRecovery now broadcasts the terminal banner before its bookkeeping storage writes (_markRecoveredSubmissionInterrupted / _recordTerminalChatStatus), so a failing write can't suppress the user-visible terminalization.
• _exhaustRecoveryGiveUp's incident read/write are best-effort (try/catch): a storage failure during give-up degrades to "maybe a second banner" rather than "silently drop the turn".

---

Bug 3 — one-shot alarm killed by a "script upgraded" supersede is not re-run

Root cause

The deploy-survival mechanism hinges on one matcher. Agent._executeScheduleCallback only defers (re-throws → preserves the one-shot row → the platform re-runs it on the fresh isolate under the at-least-once alarm guarantee) when the error matches isDurableObjectCodeUpdateReset, whose regex was /reset because its code was updated/i. The platform surfaces the same superseded-isolate failure class as a second message — "This script has been upgraded. Please send a new request to connect to the new version." (a stub/connection to a superseded script) — which fell through to the swallow-and-delete branch.

For a submitMessages turn, the driver is a one-shot drain alarm (_scheduleSubmissionDrain → this.schedule(0, "_drainThinkSubmissions", …)). When it dies with "script upgraded", the row is deleted and the submission is left with no driver: no alarm, and inbound reconnects are canceled during the bounce, so onStart (which would run _recoverSubmissionsOnStart → _startSubmissionDrain to re-drive it) never fires. The turn sits orphaned for minutes until something unrelated wakes the DO — an indefinite user-facing spinner with no terminal event. A normal deploy bounce surfaces as "reset because its code was updated" and recovers, which is exactly why the two looked inconsistent in the field.

Fix

Broaden the superseded-isolate matcher to recognize both messages, so either causes the one-shot row to be preserved and re-run on the fresh isolate. Applied in lockstep to:

• isDurableObjectCodeUpdateReset in the base Agent scheduler (packages/agents/src/index.ts) — the primary path (the submission drain / recovery one-shot alarm). This also benefits @cloudflare/ai-chat, which shares the scheduler.
• _isDeployCodeUpdateReset in Think — so a supersede thrown inside a _chatRecoveryContinue / _chatRecoveryRetry callback re-throws (defers + re-runs) instead of terminalizing via onExhausted (correcting Bug 2's classification for this class).

"Network connection lost." is deliberately not treated as a supersede: it is a connection error that may succeed on in-process retry (gated by the CF retryable property via isErrorRetryable), not an isolate replacement, so it stays on the normal retry path.

---

Relationship to #1657 (complementary, not overlapping)

Both fix the same customer-facing symptom — the default repair message "The tool call was interrupted before a result was recorded." — and both live in packages/think/src/think.ts around the _pendingInteractionPromise continuation barrier, but they address different root causes in non-overlapping code:

• fix(think): serialize parallel client-tool result/approval applies (#1649 follow-up) #1657 (merged): client tools in the normal / auto-continuation path — (a) parallel applies clobbering each other (_enqueueInteractionApply serialization) and (b) a mid-stream client result dropped before persist (_streamingAssistant). It makes the client result land so the repair backstop doesn't spuriously error it.
• This PR: server tools after deploy/eviction in the chat-recovery path — the orphan deadlocks waitUntilStable, a thrown recovery callback seals silently, and a "script upgraded" kill orphans the drain alarm.

They reinforce each other: #1657 reduces spurious client-tool input-available orphans; this PR ensures a genuinely-dead server orphan no longer wedges recovery. Rebased cleanly on top of #1657.

---

Tests

• think — predicate (waitUntilStable suite): a client-tool input-available orphan stays pending (SPA can replay); an approval-requested orphan stays pending; a server-tool input-available orphan does not block stability and waitUntilStable returns true.
• think — throw handling: a non-reset throw in a recovery callback terminalizes (fires onExhausted with reason recovery_error, delivers the banner, seals the incident exhausted) and does not re-throw; a code-update reset and a "script has been upgraded" supersede each re-throw (defer + re-run) and do not terminalize.
• agents — scheduler (schedule.test.ts): a one-shot callback throwing "reset because its code was updated" or "This script has been upgraded…" preserves the row and rejects alarm() (deferred); an ordinary error and "Network connection lost." swallow + delete the row.

Verification

• tsc --noEmit for agents and think — clean
• oxlint (agents + think src) — 0 warnings / 0 errors; oxfmt — clean
• agents — 1295 passed (full workers suite); @cloudflare/think — 493 passed

Changeset

patch for agents (Bug 3 scheduler fix) and @cloudflare/think (Bugs 1 & 2).

Behavior note

A thrown recovery callback now emits chat:recovery:exhausted (reason recovery_error) instead of chat:recovery:failed, and the recovered submission's error_message is the configured terminalMessage (the raw error is logged). For observability bridges keyed on chat:recovery:* this is more accurate (the turn is terminalized, not left ambiguously failed).

Follow-ups (out of scope)

• Mirror Bugs 1 & 2 into @cloudflare/ai-chat (duplicated recovery layer); note ai-chat has no transcript-repair pass, so Bug 1 there needs the predicate change plus a repair step. (Bug 3's base-scheduler fix already covers ai-chat.)
• Optional: surface the underlying error on ChatRecoveryExhaustedContext (additive API) if observability bridges want the cause string, not just the recovery_error category.

Made with Cursor

---

[changeset-bot]

🦋 Changeset detected

Latest commit: a26ef02

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
agents	Patch
@cloudflare/think	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1659

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1659

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1659

hono-agents

npm i https://pkg.pr.new/hono-agents@1659

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1659

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1659

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1659

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1659

commit: a26ef02

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.