Think recovery skips an interrupted opened stream with no assistant content #2276

Activity

1. ben-reitz

   self-assigned this

   on Sep 16, 2026
2. antony4859 commented on Sep 17, 2026

   

   antony4859

   on Sep 17, 2026

   Author

   More actions

   Follow-up: standalone reproduction, related fixes, and our current workaround

   We reproduced these failures in a standalone Think subclass, independently of our application and Convex. We use a scripted model, durable submitMessages() submissions, and Miniflare setOptions() restarts over the same persisted SQLite. The reproduction does not modify SDK tables or manually create recovery jobs.

   The empty-stream portion of this issue overlaps #2105. Thank you for the correction in #2256. That merged after the published 0.18.0 release. The remaining pending-retry recognition problem deserves separate attention within this issue.

   Exact environments

   • Think 0.17.0 with Agents 0.22.0.
   • Think 0.18.0 with Agents 0.23.0.
   • Node 24.18.0, AI SDK 7.0.66, Miniflare 5.20260811.1-alpha, Wrangler 4.111.0.
   • Fresh npm installs with locked dependencies. No external model calls or credentials.

   The harness applies a separate lazy-init workaround for chat@4.39.0 creating an AbortController at module scope, which workerd rejects. This workaround is identical in the stock-Think and patched-Think runs; no Think recovery code is modified in the stock runs.

   Failure 1: an empty opened stream is skipped

   1. Submit a user message using submitMessages() with a stable submission ID.
   2. Have the model open its response stream without emitting assistant content.
   3. Restart the Worker while retaining storage.
   4. Let the model answer on recovery.

   Expected: retry the same saved user prompt and complete the submission.

   Observed on published 0.17.0 and 0.18.0: status: "skipped", error: "Recovery skipped.", and the transcript still contains only the prompt.

   _recoverablePreStreamUserId rejects retry when streamId exists even though the partial is empty. Recovery selects continuation, and continueLastTurn() skips because the latest leaf is a user message. #2256 removes that classifier condition on main.

   Failure 2: startup overlooks a legitimate pending retry

   On 0.17, restart during beforeTurn preparation before the first model call. Native recovery schedules _chatRecoveryRetry, but startup reconciliation fails the submission. On 0.18, a second restart while the native recovery retry callback is active demonstrates the corresponding Tasks lookup problem.

   Expected: the matching recovery retry keeps the submission running until recovery ends.

   Observed:

   Submission was interrupted after messages were applied.
   

   _hasScheduledRecoveredContinuation recognizes only _chatRecoveryContinue. A native _chatRecoveryRetry with the same recoveredRequestId is excluded. In 0.18 this restriction exists in both the Tasks metadata predicate and the legacy schedule fallback.

   The restriction is still present in the main snapshot we inspected, commit 9efee5f0c23d6a217fe4ba5e73c078e64e9cd28c:
   

   agents/packages/think/src/think.ts

   Lines 11250 to 11283 in 9efee5f

   	private async _hasScheduledRecoveredContinuation(
   	requestId: string
   	): Promise<boolean> {
   	const recoveryRuns = await this.tasks.list({
   	definition: CHAT_RECOVERY_TASK_NAME,
   	status: ["pending", "running", "waiting"],
   	limit: Number.MAX_SAFE_INTEGER
   	});
   	if (
   	recoveryRuns.some(
   	(run) =>
   	run.metadata?.callback === "_chatRecoveryContinue" &&
   	run.metadata.recoveredRequestId === requestId
   	)
   	) {
   	return true;
   	}
   	// Dynamic-agent recovery still uses root-owned routed schedules until
   	// Tasks can mirror a child run's wake to its alarm owner.
   	return (await this.listSchedules()).some((schedule) => {
   	if (schedule.callback !== "_chatRecoveryContinue") return false;
   	const payload: unknown = schedule.payload;
   	return (
   	payload !== null &&
   	typeof payload === "object" &&
   	"recoveredRequestId" in payload &&
   	(payload as { recoveredRequestId?: unknown }).recoveredRequestId ===
   	requestId
   	);
   	});
   	}
   	// ── Programmatic API ───────────────────────────────────────────

   The callback-window scenario overrides the declared, underscore-prefixed _chatRecoveryRetry method solely to delay the retry the SDK scheduled. It then delegates to super. The separate active-turn scenario below uses the ordinary beforeTurn hook.

   Additional 0.18 case: an answer appears under an errored submission

   We also restarted after the retry callback handed execution to its new turn:

   1. Park the original turn in beforeTurn.
   2. Restart and let the SDK begin its retry.
   3. Park the retried turn in its own beforeTurn.
   4. Restart again with a completing model.

   The original submission becomes error. Four seconds later, the transcript contains the answer while the submission still reports error. This happens with published 0.18 and remains after the three targeted edits described below.

   Our source-based interpretation is that the recovery Task has completed its handoff, while the active chat-turn Task has a fresh request ID. The original submission's request ID no longer locates that execution. The status/answer inconsistency is observed; the identity explanation is an inference we would appreciate help confirming.

   Current workaround and results

   We currently pin Think 0.17.0 and carry a two-line pnpm patch in our fix PR:

   - WHERE callback = '_chatRecoveryContinue'
   + WHERE callback IN ('_chatRecoveryContinue', '_chatRecoveryRetry')

   The second edit removes streamId from _recoverablePreStreamUserId's exclusion condition, retaining the empty-partial, latest-user, continuation, and terminal-stream checks. Both edits are needed on 0.17: changing classification alone can create a retry that startup immediately invalidates.

   For 0.18 we tested the classifier correction plus recognizing Retry in both Tasks and legacy schedule predicates.

   Scenario	Stock 0.17	Patched 0.17	Stock 0.18	Patched 0.18
   Normal completion	completed	completed	completed	completed
   Empty stream, restart	skipped	completed	skipped	completed
   Retry callback interrupted before handoff	error	completed	error	completed
   Retried turn interrupted after handoff	error at first recovery	completed	error; answer later	error; answer later

   The 0.18 correction is therefore incomplete. We are keeping the 0.18 upgrade separate. Its storage migration also needs its own validation.

   Our production agent class separately passed seven local restart scenarios with the 0.17 patch, including partial-response preservation, Stop, stale retry, and preservation of a completed read-tool result. This does not establish exactly-once behavior for interrupted product writes or verify a live Cloudflare deployment.

   Application impact and related issues

   Our application stores product turn state in Convex and delivers settlement through onChatResponse or recovery exhaustion. A skipped execution can bypass normal settlement and leave the UI waiting for our twelve-minute abandonment sweep. That delay is our application behavior; the incorrect submission recovery is reproducible without our integration. We also need to harden terminal-outcome reconciliation on our side.

   • refactor(chat): add shared durable turn-status/result record (N8) #1652 directly describes premature submission terminalization during active recovery and the broader durable-outcome design.
   • Think: onChatResponse never fires if the DO resets between persisting the assistant message and running the hook #2266 describes a separate crash window after assistant persistence but before onChatResponse. This patch does not fix it.
   • Think schedules _chatRecoveryContinue for pre-first-chunk stalls, causing recovery to skip silently #1941 concerns an adjacent stall-watchdog continuation path.
   • Structured workflow turn interrupted mid-stream recovers as skipped instead of completing with output #1727 concerns structured workflow output recovery; we have not verified that path here.

   Would you prefer to keep pending-retry recognition in this issue and open a separate report for the 0.18 post-handoff case? Could retry recognition ship alongside the pending #2256 release, or be backported to 0.17? We have a standalone version-locked reproduction and JSON traces prepared for sharing.
3. ben-reitz

   added a commit that references this issue on Sep 18, 2026

   fix(think): preserve scheduled retries across restarts (#2297)

   ...

   0fd3c8b
4. ben-reitz

   closed this as completedin #2297on Sep 18, 2026
5. ben-reitz commented on Sep 18, 2026

   

   ben-reitz

   on Sep 18, 2026

   Last edited by ben-reitz

   Contributor

   More actions

   Aiming to get a new release out with these fixes this afternoon @antony4859 !

   This is now released.