fix(think): preserve scheduled retries across restarts by ben-reitz · Pull Request #2297 · cloudflare/agents

ben-reitz · 2026-09-17T22:25:19Z

Problem

After a restart, Think could mark a submission as failed even though its recovery was still in progress — or had already succeeded. Review found the reported bug plus adjacent gaps, all breaking the same invariant: a running submission must always be resolvable to live or terminal evidence at startup.

Startup checked for scheduled continuations but overlooked scheduled retries (the reported bug).
Recovery handed off to its replacement turn before that turn was durably recorded, and the replacement ran under a new request ID that startup couldn't match.
A finished turn's recovery task could disappear before the submission ledger was updated — leaving no evidence of the turn's outcome, and letting a newly starting turn reclaim what evidence remained.
Facet-hosted recoveries never signaled handoff, holding the recovery step for the full model turn against its 15-minute timeout.
The recovery lookup matched submission and request identities without precedence, so a collision could touch the wrong submission.
Reconciliation inferred the outcome from stream state — but streams also settle as completed on abort and on overflow retry, and structured output only lived in memory. A crash in the settlement window could report a cancelled submission as completed, or emit a completed workflow event with output: undefined.

Fix

Recognize both kinds of recovery work at startup; keep the submission bound to whichever side currently owns it across the handoff (the handoff signal is scoped to the intended successor's call chain, on both the Tasks and facet-fiber paths); and settle the ledger from a durable record of the turn's actual outcome without rerunning the turn or repeating callbacks.

Each turn's terminal outcome (completed, aborted, retry, or error) and structured output are recorded on the submission row in the same transaction that settles its stream, so restarts settle from recorded fact rather than inference: aborts stay aborted, recovered workflow completions keep their exact output, an overflow retry segment is never mistaken for an answer, and the record survives unrelated turns reclaiming stream rows. Terminal-stream inference remains only as a fallback for rows written before this change. Exact submission IDs take precedence over request-ID fallback, and new recovery payloads carry the stable submission identity (old payloads still match).

The agents change is deliberately tiny (changeset: patch): expose ResumableStream.pendingCutoverId, and make finalizePending clear its pending marker only after settlement succeeds so a failed settlement stays retryable — matching cutover. An earlier revision added a retained-stream lifecycle to ResumableStream; the outcome record made it redundant, so it was removed before release rather than shipped as API surface.

 on restart
- only a scheduled continuation keeps the submission running
+ scheduled retries, the accepted replacement turn, and the recorded
+ turn outcome all keep or settle the submission correctly

Scope note

This fixes more than the original startup-detection gap: closing only item 1 still failed deterministic interruption tests for the reported symptom (see the reporter's third scenario in #2276, whose request-ID analysis matches item 2 exactly). The commits land on clean boundaries — f620ff54 (item 1), 2984fba5 + dfa76d65 (item 2), 91ee77ba (items 3–5), ea424ebe (item 6), 0f7d380f (consolidation onto the outcome record + finalizePending retry fix) — and can be split on request.

Validation: Deterministic red-first tests for every window above (including foreign-turn interference, foreign-stream reclamation with a real process restart, abort/output crash windows, facet handoff, and ID collisions), full Think (936) and agents suites, submission-recovery e2e (9), build, and pnpm run check — all passing with retries disabled.

Fixes #2276

Startup treated only continuation callbacks as outstanding recovery, so an already-scheduled retry could lose its running submission after restart. Recognize both callbacks at the existing submission-recovery seam for Tasks and legacy schedules. Cover pending retries, request isolation, terminal Tasks, cancellation, empty opened streams, and persisted restart completion. Refs #2276.

Review found two remaining interruption windows: the recovery callback signaled handoff before the successor turn was durably accepted, and a non-retained successor could settle before ledger completion. Rebind the running submission to the accepted successor before handoff, retain terminal stream evidence for running submissions, and settle the ledger from that evidence at startup without rerunning the turn. Replace the e2e table-surgery staging with inert seeded evidence driven through two real restarts so production owns retry scheduling and linkage.

Review found the acceptance callback lived on the agent instance, so any concurrent root chat turn admitted during the successor's pre-admission awaits could claim the handoff, rebind the submission to a foreign request, and release the recovery Task early. Hold the callback in successor-scoped async context so only the intended turn's chain can signal acceptance.

…ollision gaps Review of the handoff work found three follow-ups: facet-hosted recoveries never signaled acceptance, so the recovery step awaited the whole model turn against its 15-minute timeout; stream reclamation deleted the terminal evidence retained for running submissions as soon as any new turn started; and the recovery lookup matched submission and request identities without precedence, letting a collision touch the wrong submission. ResumableStream now owns a durable retained-row lifecycle (retain at settlement, skip at reclaim, release when settled) and Think releases pins at every ledger settlement and on startup. Facet successors signal acceptance once their fiber is durably created. Exact submission identity takes precedence over request-id fallback, and new recovery payloads carry the stable submission id.

changeset-bot · 2026-09-17T22:25:24Z

🦋 Changeset detected

Latest commit: 0f7d380

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
agents	Patch
@cloudflare/think	Patch
@cloudflare/agent-think	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2026-09-17T22:39:11Z

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2297

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2297

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2297

hono-agents

npm i https://pkg.pr.new/hono-agents@2297

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2297

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2297

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2297

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2297

commit: 0f7d380

Startup and recovery reconciliation inferred a running submission's outcome from stream lifecycle state, but streams also settle as completed on abort and on overflow retry, and the structured output only ever lived in memory. A crash between stream settlement and the ledger write could report a cancelled submission as completed, complete an overflow segment whose answer was never persisted, or emit a completed workflow event with an undefined output. Record the turn's terminal outcome ('completed' | 'aborted' | 'retry') and its structured output on the submission row in the same transaction that settles the stream. Reconciliation settles from that recorded fact, passing the recorded output through to workflow notifications; stream-status inference remains only as a fallback for legacy rows, and a retry stamp is never completion evidence.

agent-think · 2026-09-17T23:17:56Z

🟡 agents import sizes

Measured 343 runtime imports as minified bundles. The primary size is gzip; raw minified size is included for diagnosis. An existing import growing by more than 10% is marked red. This report is informational.

Red	Yellow	Green	Unchanged	New	Removed
0	1	0	342	0	0

Compared 16b68560 with 0f7d380f. Open workflow run.

Changed imports (1)

Status	Import	Base gzip	Head gzip	Delta
🟡	`agents/chat#ResumableStream`	5.1 KiB	5.1 KiB	+2 B (+0.04%)

All 343 current runtime imports

Status	Import	Gzip	Raw minified
⚪	`agents#__DO_NOT_USE_WILL_BREAK__agentContext`	261.1 KiB	1141.0 KiB
⚪	`agents#__DO_NOT_USE_WILL_BREAK__withInvocationScope`	261.1 KiB	1141.0 KiB
⚪	`agents#Agent`	261.1 KiB	1141.0 KiB
⚪	`agents#AGENT_TOOL_MILESTONE_PART`	261.1 KiB	1141.0 KiB
⚪	`agents#AGENT_TOOL_PROGRESS_PART`	261.1 KiB	1141.0 KiB
⚪	`agents#buildAgentPath`	261.7 KiB	1143.3 KiB
⚪	`agents#buildAgentUrl`	261.8 KiB	1143.7 KiB
⚪	`agents#callable`	261.1 KiB	1141.0 KiB
⚪	`agents#camelCaseToKebabCase`	261.1 KiB	1141.0 KiB
⚪	`agents#createHeaderBasedEmailResolver`	261.3 KiB	1141.4 KiB
⚪	`agents#DEFAULT_AGENT_STATIC_OPTIONS`	261.1 KiB	1141.0 KiB
⚪	`agents#DurableObjectOAuthClientProvider`	261.1 KiB	1141.0 KiB
⚪	`agents#getAgentByName`	261.1 KiB	1141.0 KiB
⚪	`agents#getCurrentAgent`	261.1 KiB	1141.0 KiB
⚪	`agents#getSubAgentByName`	261.4 KiB	1141.6 KiB
⚪	`agents#isDurableObjectCodeUpdateReset`	261.1 KiB	1141.0 KiB
⚪	`agents#isDurableObjectMemoryLimitReset`	261.1 KiB	1141.0 KiB
⚪	`agents#isDurableObjectStorageReset`	261.1 KiB	1141.0 KiB
⚪	`agents#isPlatformTransientError`	261.1 KiB	1141.0 KiB
⚪	`agents#MCP_SERVER_ID_MAX_LENGTH`	261.1 KiB	1141.0 KiB
⚪	`agents#MessageType`	261.2 KiB	1141.3 KiB
⚪	`agents#normalizeServerId`	261.1 KiB	1141.0 KiB
⚪	`agents#parseSubAgentPath`	261.1 KiB	1141.0 KiB
⚪	`agents#routeAgentEmail`	261.4 KiB	1141.7 KiB
⚪	`agents#routeAgentRequest`	261.7 KiB	1142.9 KiB
⚪	`agents#routeSubAgentRequest`	261.3 KiB	1141.5 KiB
⚪	`agents#SqlError`	261.1 KiB	1141.0 KiB
⚪	`agents#StreamingResponse`	261.1 KiB	1141.0 KiB
⚪	`agents#SUB_PREFIX`	261.1 KiB	1141.0 KiB
⚪	`agents#unstable_callable`	261.2 KiB	1141.2 KiB
⚪	`agents/agent-tools#agentTool`	112.5 KiB	538.2 KiB
⚪	`agents/browser#BrowserConnector`	50.5 KiB	176.6 KiB
⚪	`agents/browser#browserContent`	36.3 KiB	127.4 KiB
⚪	`agents/browser#browserExtract`	36.3 KiB	127.4 KiB
⚪	`agents/browser#browserLinks`	36.3 KiB	127.4 KiB
⚪	`agents/browser#browserMarkdown`	36.3 KiB	127.4 KiB
⚪	`agents/browser#browserPdf`	36.3 KiB	127.3 KiB
⚪	`agents/browser#BrowserRenderingError`	36.0 KiB	126.7 KiB
⚪	`agents/browser#browserScrape`	36.3 KiB	127.4 KiB
⚪	`agents/browser#browserScreenshot`	36.3 KiB	127.3 KiB
⚪	`agents/browser#browserSnapshot`	36.3 KiB	127.4 KiB
⚪	`agents/browser#CdpSession`	37.2 KiB	129.8 KiB
⚪	`agents/browser#CodemodeRuntime`	39.6 KiB	139.0 KiB
⚪	`agents/browser#connectBrowser`	37.8 KiB	131.4 KiB
⚪	`agents/browser#connectBrowserSession`	37.5 KiB	130.4 KiB
⚪	`agents/browser#connectUrl`	37.6 KiB	130.5 KiB
⚪	`agents/browser#createBrowserSession`	36.3 KiB	127.5 KiB
⚪	`agents/browser#DEFAULT_EXEC_SWEEP_IDLE_MS`	36.0 KiB	126.6 KiB
⚪	`agents/browser#DEFAULT_SWEEP_IDLE_MS`	36.0 KiB	126.6 KiB
⚪	`agents/browser#deleteBrowserSession`	36.1 KiB	126.9 KiB
⚪	`agents/browser#DurableBrowserSessionStore`	36.4 KiB	127.6 KiB
⚪	`agents/browser#getBrowserRecording`	36.2 KiB	127.1 KiB
⚪	`agents/browser#listBrowserTargets`	36.1 KiB	126.9 KiB
⚪	`agents/browser#loadCdpSpec`	36.6 KiB	128.3 KiB
⚪	`agents/browser#runQuickAction`	36.0 KiB	126.6 KiB
⚪	`agents/browser/ai#createBrowserRuntime`	147.0 KiB	632.8 KiB
⚪	`agents/browser/ai#createBrowserTools`	147.0 KiB	632.9 KiB
⚪	`agents/browser/ai#createQuickActionTools`	122.5 KiB	554.3 KiB
⚪	`agents/browser/tanstack-ai#createBrowserTools`	162.7 KiB	701.7 KiB
⚪	`agents/channels#ChannelHost`	3.3 KiB	9.1 KiB
⚪	`agents/channels#consumeChunks`	237 B	321 B
⚪	`agents/channels#createUserIdentityStore`	1.3 KiB	3.2 KiB
⚪	`agents/channels#fallback`	155 B	160 B
⚪	`agents/channels#fallbackChannel`	893 B	1.9 KiB
⚪	`agents/channels#fanout`	151 B	156 B
⚪	`agents/channels#fanoutChannel`	875 B	1.9 KiB
⚪	`agents/channels#identityKey`	178 B	223 B
⚪	`agents/channels#isChannelMessageSurface`	271 B	452 B
⚪	`agents/channels#linkChannelIdentities`	113 B	99 B
⚪	`agents/channels#matchesPath`	110 B	96 B
⚪	`agents/channels#routes`	307 B	507 B
⚪	`agents/channels#UserIdentityConflictError`	248 B	335 B
⚪	`agents/channels/ai-sdk#createSendMessageTool`	112.2 KiB	537.4 KiB
⚪	`agents/channels/ai-sdk#toChannelChunks`	112.5 KiB	538.4 KiB
⚪	`agents/channels/email#email`	23.5 KiB	75.4 KiB
⚪	`agents/channels/email#inboundEmail`	22.3 KiB	72.3 KiB
⚪	`agents/channels/slack#slack`	5.5 KiB	14.6 KiB
⚪	`agents/channels/slack#slackWebhook`	2.2 KiB	5.1 KiB
⚪	`agents/channels/tanstack-ai#createSendMessageTool`	16.2 KiB	68.4 KiB
⚪	`agents/channels/telegram#telegram`	3.8 KiB	10.3 KiB
⚪	`agents/channels/telegram#telegramWebhook`	1.6 KiB	3.6 KiB
⚪	`agents/channels/voice#browserVoice`	1010 B	2.2 KiB
⚪	`agents/chat#AbortRegistry`	2.5 KiB	8.9 KiB
⚪	`agents/chat#AGENT_TOOL_STREAM_PROGRESS_BUMP_THROTTLE_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#AgentToolProgressEmitter`	2.7 KiB	9.5 KiB
⚪	`agents/chat#AgentToolStreamProgressThrottle`	2.4 KiB	8.3 KiB
⚪	`agents/chat#aiSdkRecoveryCodec`	2.3 KiB	8.2 KiB
⚪	`agents/chat#applyAgentToolEvent`	3.2 KiB	11.0 KiB
⚪	`agents/chat#applyChunkToParts`	2.3 KiB	8.2 KiB
⚪	`agents/chat#applyToolUpdate`	2.4 KiB	8.4 KiB
⚪	`agents/chat#AutoContinuationController`	2.3 KiB	8.2 KiB
⚪	`agents/chat#awaitWithDeadline`	2.4 KiB	8.4 KiB
⚪	`agents/chat#broadcastTransition`	3.2 KiB	11.4 KiB
⚪	`agents/chat#buildChatRecoveringFrame`	2.4 KiB	8.4 KiB
⚪	`agents/chat#buildInClauseStrings`	2.4 KiB	8.4 KiB
⚪	`agents/chat#bumpChatRecoveryProgress`	2.4 KiB	8.3 KiB
⚪	`agents/chat#byteLength`	2.5 KiB	8.5 KiB
⚪	`agents/chat#CHAT_LAST_TERMINAL_KEY`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_MESSAGE_TYPES`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERING_FLAG_TTL_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERING_KEY`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_ALARM_DEBOUNCE_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_INCIDENT_KEY_PREFIX`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_INCIDENT_TTL_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_PROGRESS_KEY`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_STABLE_RETRY_DELAY_SECONDS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_RECOVERY_TASK_NAME`	2.3 KiB	8.2 KiB
⚪	`agents/chat#CHAT_STREAM_PROGRESS_CREDIT_THROTTLE_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#ChatRecoveryEngine`	4.4 KiB	15.3 KiB
⚪	`agents/chat#chatRecoveryTaskRunOptions`	2.4 KiB	8.6 KiB
⚪	`agents/chat#ChatStreamStalledError`	2.4 KiB	8.3 KiB
⚪	`agents/chat#classifyAgentToolChildRecovery`	2.4 KiB	8.5 KiB
⚪	`agents/chat#clearChatTerminal`	2.3 KiB	8.3 KiB
⚪	`agents/chat#clientResolvableToolNames`	2.3 KiB	8.3 KiB
⚪	`agents/chat#ContinuationState`	2.7 KiB	9.8 KiB
⚪	`agents/chat#createAgentToolEventState`	2.3 KiB	8.3 KiB
⚪	`agents/chat#createChatFiberSnapshot`	2.5 KiB	8.6 KiB
⚪	`agents/chat#createChatRecoveryTaskDefinition`	2.6 KiB	9.0 KiB
⚪	`agents/chat#createChatStreams`	6.4 KiB	22.1 KiB
⚪	`agents/chat#createChatTurnTaskDefinition`	2.7 KiB	8.9 KiB
⚪	`agents/chat#createToolsFromClientSchemas`	114.3 KiB	545.4 KiB
⚪	`agents/chat#crossMessageToolResultUpdate`	2.4 KiB	8.6 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_MAX_ATTEMPTS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_MAX_OOM_RETRIES`	2.3 KiB	8.2 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_MAX_WORK`	2.3 KiB	8.2 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_NO_PROGRESS_TIMEOUT_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_STABLE_TIMEOUT_MS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#DEFAULT_CHAT_RECOVERY_TERMINAL_MESSAGE`	2.4 KiB	8.3 KiB
⚪	`agents/chat#dispatchChatRecoveryToHandoff`	3.0 KiB	9.9 KiB
⚪	`agents/chat#drainInteractionApplies`	2.3 KiB	8.3 KiB
⚪	`agents/chat#enforceRowSizeLimit`	3.5 KiB	11.2 KiB
⚪	`agents/chat#hasIncompleteToolBatch`	2.4 KiB	8.6 KiB
⚪	`agents/chat#interceptAgentToolBroadcast`	2.5 KiB	8.7 KiB
⚪	`agents/chat#isPlatformFailure`	2.6 KiB	8.9 KiB
⚪	`agents/chat#isReplayChunk`	2.4 KiB	8.7 KiB
⚪	`agents/chat#iterateWithStallWatchdog`	2.6 KiB	8.8 KiB
⚪	`agents/chat#KV_DELETE_MAX_KEYS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#listActiveChatRecoveryIncidents`	2.4 KiB	8.4 KiB
⚪	`agents/chat#MAX_BOUND_PARAMS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#MessageType`	2.4 KiB	9.0 KiB
⚪	`agents/chat#normalizeToolInput`	2.3 KiB	8.2 KiB
⚪	`agents/chat#parseProtocolMessage`	2.5 KiB	9.0 KiB
⚪	`agents/chat#partAwaitsClientInteraction`	2.4 KiB	8.6 KiB
⚪	`agents/chat#pausedExecutionUpdate`	2.4 KiB	8.4 KiB
⚪	`agents/chat#pendingChatTerminal`	2.3 KiB	8.3 KiB
⚪	`agents/chat#persistReconstructedOrphan`	3.1 KiB	11.0 KiB
⚪	`agents/chat#PreStreamTurns`	2.6 KiB	9.3 KiB
⚪	`agents/chat#readChatRecoveryProgress`	2.3 KiB	8.3 KiB
⚪	`agents/chat#reconcileMessages`	2.8 KiB	9.6 KiB
⚪	`agents/chat#reconcileOrphanPartial`	2.4 KiB	8.5 KiB
⚪	`agents/chat#recordChatTerminal`	2.4 KiB	8.3 KiB
⚪	`agents/chat#repairInterruptedToolParts`	2.6 KiB	9.1 KiB
⚪	`agents/chat#resolveChatRecoveryConfig`	2.6 KiB	8.9 KiB
⚪	`agents/chat#resolveToolMergeId`	2.4 KiB	8.5 KiB
🟡	`agents/chat#ResumableStream`	5.1 KiB	16.8 KiB
⚪	`agents/chat#ResumeHandshake`	3.0 KiB	10.4 KiB
⚪	`agents/chat#ROW_MAX_BYTES`	2.3 KiB	8.2 KiB
⚪	`agents/chat#runChatRecoveryExhaustion`	2.6 KiB	8.9 KiB
⚪	`agents/chat#sanitizeMessage`	2.5 KiB	8.8 KiB
⚪	`agents/chat#sendIfOpen`	2.4 KiB	8.4 KiB
⚪	`agents/chat#setChatRecovering`	2.5 KiB	8.5 KiB
⚪	`agents/chat#shouldCreditStreamProgress`	2.4 KiB	8.3 KiB
⚪	`agents/chat#STREAM_RESUME_NONE_REASONS`	2.3 KiB	8.2 KiB
⚪	`agents/chat#StreamAccumulator`	2.9 KiB	10.7 KiB
⚪	`agents/chat#StreamProgressCreditThrottle`	2.4 KiB	8.3 KiB
⚪	`agents/chat#SubmitConcurrencyController`	2.9 KiB	10.3 KiB
⚪	`agents/chat#sweepStaleChatRecoveryIncidents`	2.4 KiB	8.5 KiB
⚪	`agents/chat#TextSegmentJoiner`	2.7 KiB	9.2 KiB
⚪	`agents/chat#TIMED_OUT`	2.3 KiB	8.2 KiB
⚪	`agents/chat#toolApprovalUpdate`	2.4 KiB	8.5 KiB
⚪	`agents/chat#toolPartHasSettledResult`	2.3 KiB	8.4 KiB
⚪	`agents/chat#toolResultUpdate`	2.4 KiB	8.5 KiB
⚪	`agents/chat#truncateOlderMessages`	3.2 KiB	10.4 KiB
⚪	`agents/chat#TurnQueue`	2.6 KiB	9.2 KiB
⚪	`agents/chat#unwrapChatFiberSnapshot`	2.4 KiB	8.5 KiB
⚪	`agents/chat#wrapChatFiberSnapshot`	2.3 KiB	8.2 KiB
⚪	`agents/chat-sdk#ChatSdkStateAdapter`	263.5 KiB	1152.6 KiB
⚪	`agents/chat-sdk#ChatSdkStateAgent`	262.8 KiB	1150.0 KiB
⚪	`agents/chat-sdk#createChatSdkState`	263.5 KiB	1152.6 KiB
⚪	`agents/chat-sdk#defaultKeyShard`	261.2 KiB	1141.2 KiB
⚪	`agents/chat-sdk#defaultThreadShard`	261.1 KiB	1141.0 KiB
⚪	`agents/chat/react#detectToolsRequiringConfirmation`	3.3 KiB	8.3 KiB
⚪	`agents/chat/react#extractClientToolSchemas`	3.2 KiB	8.3 KiB
⚪	`agents/chat/react#getAgentMessages`	3.4 KiB	8.6 KiB
⚪	`agents/chat/react#getToolApproval`	3.1 KiB	8.0 KiB
⚪	`agents/chat/react#getToolCallId`	3.1 KiB	8.0 KiB
⚪	`agents/chat/react#getToolInput`	3.1 KiB	8.0 KiB
⚪	`agents/chat/react#getToolOutput`	3.1 KiB	8.0 KiB
⚪	`agents/chat/react#getToolPartState`	3.2 KiB	8.2 KiB
⚪	`agents/chat/react#useAgentChat`	132.9 KiB	609.7 KiB
⚪	`agents/chat/react#WebSocketChatTransport`	5.7 KiB	17.1 KiB
⚪	`agents/chat/transport#WebSocketChatTransport`	2.8 KiB	9.2 KiB
⚪	`agents/client#AgentClient`	21.8 KiB	75.2 KiB
⚪	`agents/client#AgentConnectionError`	16.3 KiB	57.8 KiB
⚪	`agents/client#agentFetch`	19.7 KiB	69.0 KiB
⚪	`agents/client#createStubProxy`	16.4 KiB	57.8 KiB
⚪	`agents/client#DEFAULT_CALL_TIMEOUT_MS`	16.2 KiB	57.6 KiB
⚪	`agents/client#isTerminalCloseEvent`	16.3 KiB	57.6 KiB
⚪	`agents/client#nativeCall`	16.5 KiB	58.3 KiB
⚪	`agents/client#NativeCallQueue`	16.7 KiB	58.9 KiB
⚪	`agents/client#splitCallOptions`	16.3 KiB	57.7 KiB
⚪	`agents/context#AgentContextProvider`	412 B	792 B
⚪	`agents/context#AgentSearchProvider`	640 B	1.3 KiB
⚪	`agents/context#ContextBlocks`	87.4 KiB	429.7 KiB
⚪	`agents/email#createAddressBasedEmailResolver`	193 B	227 B
⚪	`agents/email#createCatchAllEmailResolver`	110 B	97 B
⚪	`agents/email#createHeaderBasedEmailResolver`	334 B	492 B
⚪	`agents/email#createSecureReplyEmailResolver`	718 B	1.3 KiB
⚪	`agents/email#DEFAULT_MAX_AGE_SECONDS`	56 B	39 B
⚪	`agents/email#isAutoReplyEmail`	201 B	249 B
⚪	`agents/email#signAgentHeaders`	424 B	812 B
⚪	`agents/experimental/webmcp#registerWebMcp`	85.2 KiB	295.8 KiB
⚪	`agents/lifecycle#getCurrentAgent`	376 B	798 B
⚪	`agents/lifecycle#Lifecycle`	8.4 KiB	26.1 KiB
⚪	`agents/lifecycle#LifecycleCapability`	497 B	999 B
⚪	`agents/mcp#createLegacyMcpHandler`	378.6 KiB	1583.3 KiB
⚪	`agents/mcp#createMcpHandler`	390.9 KiB	1628.5 KiB
⚪	`agents/mcp#DurableObjectEventStore`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#ElicitRequestSchema`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#experimental_createMcpHandler`	378.7 KiB	1583.6 KiB
⚪	`agents/mcp#getMcpAuthContext`	345.1 KiB	1441.9 KiB
⚪	`agents/mcp#MCP_SERVER_ID_MAX_LENGTH`	345.1 KiB	1441.9 KiB
⚪	`agents/mcp#McpAgent`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#normalizeServerId`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#RPC_DO_PREFIX`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#RPCClientTransport`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#RPCServerTransport`	345.1 KiB	1441.8 KiB
⚪	`agents/mcp#SSEEdgeClientTransport`	345.2 KiB	1442.1 KiB
⚪	`agents/mcp#StreamableHTTPEdgeClientTransport`	345.2 KiB	1442.1 KiB
⚪	`agents/mcp#WorkerTransport`	348.5 KiB	1458.7 KiB
⚪	`agents/mcp/client#getNamespacedData`	62.9 KiB	240.0 KiB
⚪	`agents/mcp/client#MCP_SERVER_ID_MAX_LENGTH`	62.9 KiB	239.9 KiB
⚪	`agents/mcp/client#MCPClientManager`	158.6 KiB	702.6 KiB
⚪	`agents/mcp/client#normalizeServerId`	63.0 KiB	240.2 KiB
⚪	`agents/mcp/do-oauth-client-provider#DurableObjectOAuthClientProvider`	2.1 KiB	6.6 KiB
⚪	`agents/mcp/server#createMcpHandler`	80.5 KiB	307.2 KiB
⚪	`agents/mcp/server#getMcpAuthContext`	64.0 KiB	245.5 KiB
⚪	`agents/observability#channels`	259 B	549 B
⚪	`agents/observability#genericObservability`	470 B	1.2 KiB
⚪	`agents/observability#subscribe`	324 B	668 B
⚪	`agents/observability/ai#wrapAISDK`	8.8 KiB	30.5 KiB
⚪	`agents/queue#Queue`	3.1 KiB	9.6 KiB
⚪	`agents/react#_testUtils`	19.3 KiB	66.4 KiB
⚪	`agents/react#useAgent`	27.1 KiB	89.9 KiB
⚪	`agents/react#useAgentToolEvents`	21.1 KiB	73.7 KiB
⚪	`agents/routing#getAgentByName`	795 B	1.7 KiB
⚪	`agents/routing#routeAgentRequest`	1.6 KiB	3.6 KiB
⚪	`agents/routing#RoutedAgents`	2.4 KiB	6.2 KiB
⚪	`agents/schedule#getSchedulePrompt`	85.8 KiB	424.7 KiB
⚪	`agents/schedule#scheduleSchema`	85.3 KiB	423.6 KiB
⚪	`agents/schedule#unstable_getSchedulePrompt`	85.9 KiB	424.9 KiB
⚪	`agents/schedule#unstable_scheduleSchema`	85.3 KiB	423.6 KiB
⚪	`agents/schedules#Scheduler`	6.8 KiB	22.0 KiB
⚪	`agents/schedules/parser#getSchedulePrompt`	85.8 KiB	424.7 KiB
⚪	`agents/schedules/parser#scheduleSchema`	85.3 KiB	423.6 KiB
⚪	`agents/sessions#COMPACTION_PREFIX`	147 B	160 B
⚪	`agents/sessions#createCompactFunction`	1.7 KiB	4.0 KiB
⚪	`agents/sessions#isCompactionMessage`	177 B	200 B
⚪	`agents/sessions#Session`	2.0 KiB	6.3 KiB
⚪	`agents/sessions#Sessions`	9.6 KiB	34.3 KiB
⚪	`agents/skills#fromManifest`	309.8 KiB	1084.0 KiB
⚪	`agents/skills#parseSkillFrontmatter`	328.4 KiB	1146.2 KiB
⚪	`agents/skills#parseSkillMarkdown`	328.6 KiB	1146.5 KiB
⚪	`agents/skills#r2`	330.2 KiB	1150.4 KiB
⚪	`agents/skills#runner`	369.0 KiB	1297.8 KiB
⚪	`agents/skills#SkillRegistry`	416.4 KiB	1581.7 KiB
⚪	`agents/skills/compile#compileSkillScript`	15.4 KiB	43.4 KiB
⚪	`agents/skills/compile#isCompilableSkillScript`	15.4 KiB	43.3 KiB
⚪	`agents/state#State`	1.1 KiB	2.4 KiB
⚪	`agents/streams#DEFAULT_MAX_CHUNK_BYTES`	94 B	96 B
⚪	`agents/streams#sseResponse`	857 B	1.6 KiB
⚪	`agents/streams#StreamClosedError`	175 B	212 B
⚪	`agents/streams#StreamNotFoundError`	215 B	276 B
⚪	`agents/streams#Streams`	4.3 KiB	13.9 KiB
⚪	`agents/streams#StreamSerializationError`	171 B	201 B
⚪	`agents/tasks#DuplicateTaskStepError`	328 B	463 B
⚪	`agents/tasks#MAX_SERIALIZED_BYTES`	190 B	232 B
⚪	`agents/tasks#MissingTaskDefinitionError`	358 B	536 B
⚪	`agents/tasks#NonRetryableError`	238 B	308 B
⚪	`agents/tasks#TaskReplayDivergedError`	341 B	483 B
⚪	`agents/tasks#Tasks`	8.9 KiB	31.8 KiB
⚪	`agents/tasks#TaskSerializationError`	258 B	339 B
⚪	`agents/types#MessageType`	211 B	365 B
⚪	`agents/vite#default`	353.8 KiB	1356.1 KiB
⚪	`agents/voice#addSFUTracks`	324 B	424 B
⚪	`agents/voice#createSFUSession`	255 B	306 B
⚪	`agents/voice#createSFUWebSocketAdapter`	331 B	429 B
⚪	`agents/voice#decodeVarint`	157 B	160 B
⚪	`agents/voice#downsample48kStereoTo16kMono`	238 B	324 B
⚪	`agents/voice#encodePayloadToProtobuf`	189 B	279 B
⚪	`agents/voice#encodeVarint`	129 B	122 B
⚪	`agents/voice#extractPayloadFromProtobuf`	271 B	425 B
⚪	`agents/voice#iterateText`	1.6 KiB	3.8 KiB
⚪	`agents/voice#renegotiateSFUSession`	329 B	428 B
⚪	`agents/voice#SentenceChunker`	550 B	1.1 KiB
⚪	`agents/voice#sfuFetch`	294 B	358 B
⚪	`agents/voice#upsample16kMonoTo48kStereo`	211 B	272 B
⚪	`agents/voice#VOICE_PROTOCOL_VERSION`	51 B	31 B
⚪	`agents/voice#withVoice`	9.5 KiB	32.7 KiB
⚪	`agents/voice#withVoiceInput`	4.8 KiB	16.0 KiB
⚪	`agents/voice#WorkersAIFluxSTT`	1.5 KiB	4.3 KiB
⚪	`agents/voice#WorkersAINova3STT`	1.6 KiB	4.3 KiB
⚪	`agents/voice#WorkersAITTS`	682 B	1.4 KiB
⚪	`agents/voice/client#VOICE_PROTOCOL_VERSION`	473 B	768 B
⚪	`agents/voice/client#VoiceClient`	10.1 KiB	31.6 KiB
⚪	`agents/voice/client#WebSocketVoiceTransport`	4.6 KiB	13.3 KiB
⚪	`agents/voice/errors#logVoiceError`	132 B	173 B
⚪	`agents/voice/errors#toVoiceError`	94 B	80 B
⚪	`agents/voice/errors#voiceErrorMessage`	125 B	111 B
⚪	`agents/voice/errors#VoiceProviderError`	190 B	345 B
⚪	`agents/voice/react#useVoiceAgent`	13.6 KiB	42.5 KiB
⚪	`agents/voice/react#useVoiceInput`	13.3 KiB	41.4 KiB
⚪	`agents/voice/react#WebSocketVoiceTransport`	7.4 KiB	21.3 KiB
⚪	`agents/voice/sfu#addSFUTracks`	323 B	424 B
⚪	`agents/voice/sfu#createSFUSession`	254 B	306 B
⚪	`agents/voice/sfu#createSFUWebSocketAdapter`	329 B	429 B
⚪	`agents/voice/sfu#decodeVarint`	155 B	160 B
⚪	`agents/voice/sfu#downsample48kStereoTo16kMono`	236 B	324 B
⚪	`agents/voice/sfu#encodePayloadToProtobuf`	188 B	279 B
⚪	`agents/voice/sfu#encodeVarint`	129 B	122 B
⚪	`agents/voice/sfu#extractPayloadFromProtobuf`	270 B	425 B
⚪	`agents/voice/sfu#renegotiateSFUSession`	329 B	428 B
⚪	`agents/voice/sfu#sfuFetch`	292 B	358 B
⚪	`agents/voice/sfu#upsample16kMonoTo48kStereo`	209 B	272 B
⚪	`agents/voice/text#iterateText`	1.6 KiB	3.8 KiB
⚪	`agents/voice/text#SentenceChunker`	549 B	1.1 KiB
⚪	`agents/voice/types#VOICE_PROTOCOL_VERSION`	51 B	31 B
⚪	`agents/voice/workers-ai#WorkersAIFluxSTT`	1.5 KiB	4.3 KiB
⚪	`agents/voice/workers-ai#WorkersAINova3STT`	1.6 KiB	4.3 KiB
⚪	`agents/voice/workers-ai#WorkersAITTS`	682 B	1.4 KiB
⚪	`agents/websockets#CAPNWEB_TRANSPORT_QUERY`	12.4 KiB	43.3 KiB
⚪	`agents/websockets#CAPNWEB_TRANSPORT_VALUE`	12.4 KiB	43.3 KiB
⚪	`agents/websockets#capnWebTransportUrl`	12.5 KiB	43.5 KiB
⚪	`agents/websockets#CF_NO_PROTOCOL_KEY`	12.4 KiB	43.3 KiB
⚪	`agents/websockets#CF_READONLY_KEY`	12.4 KiB	43.3 KiB
⚪	`agents/websockets#isCapnWebTransportUpgrade`	12.5 KiB	43.4 KiB
⚪	`agents/websockets#registerInternalConnectionKeys`	12.4 KiB	43.4 KiB
⚪	`agents/websockets#WebSockets`	20.0 KiB	69.3 KiB
⚪	`agents/workflows#AgentWorkflow`	262.4 KiB	1145.7 KiB
⚪	`agents/workflows#WorkflowRejectedError`	261.2 KiB	1141.2 KiB
⚪	`agents/x402#normalizeNetwork`	14.7 KiB	61.1 KiB
⚪	`agents/x402#withX402`	23.0 KiB	89.2 KiB
⚪	`agents/x402#withX402Client`	104.1 KiB	346.5 KiB

Reported by agent-think[bot].

…ention The branch carried two overlapping restart-evidence mechanisms: retained terminal stream rows (round 3) and durable outcome stamps (round 4). Stamps are the atomic recorded fact, so consolidate on them before the unreleased retention API becomes contractual: - agents: remove retain/release/listRetained and the retained metadata, reclaim exclusion, and updateMetadata plumbing. Clear the pending cutover marker only after settlement succeeds in finalizePending so a failed settlement stays retryable, matching cutover. - think: stamp result_status='error' atomically with markError so errored-turn evidence survives foreign stream reclamation without pinning rows; startup and recovered settlement read the stamp with pending recovery still taking priority. Submission streams are again discarded at cutover; the stamp is the evidence. Remove all pin bookkeeping and clear stamps on cancel/interrupt terminal writes. - tests: replace retention assertions with stamp-lifecycle assertions; new agents settlement tests cover persist rollback and retryable finalizePending.

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-09-18T10:15:35Z

+  /** Write only inside the transaction that settles this exact request's stream. */
+  private _recordSubmissionTurnResult(
+    requestId: string,
+    result: SubmissionTurnResult
+  ): void {
+    const row = this._readRunningSubmissionForRecovery(requestId);
+    if (!row || row.request_id !== requestId) return;
+    this.sql`
+      UPDATE cf_think_submissions
+      SET result_status = ${result.status},
+          output_json = ${result.status === "completed" && result.output !== undefined ? JSON.stringify(result.output) : null}
+      WHERE submission_id = ${row.submission_id} AND status = 'running'
+    `;


📝 Info: Outcome stamps replace stream pinning

The final design differs from the PR description's retained-stream API. Atomic submission stamps now preserve terminal evidence while streams remain reclaimable.

Was this helpful? React with 👍 or 👎 to provide feedback.

ben-reitz added 4 commits September 16, 2026 15:55

ben-reitz mentioned this pull request Sep 17, 2026

fix(think): preserve scheduled retries across restarts #2289

Closed

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot reviewed Sep 18, 2026

View reviewed changes

ben-reitz merged commit 0fd3c8b into main Sep 18, 2026
18 checks passed

ben-reitz deleted the fix/2276-preserve-scheduled-chat-recovery-v2 branch September 18, 2026 10:31

github-actions Bot mentioned this pull request Sep 17, 2026

Version Packages #2251

Merged

MiguelsPizza mentioned this pull request Sep 23, 2026

Refresh the vendored Agents SDK to 0.24.0 / Think 0.19.0 WebMCP-org/do-runtime#81

Merged

This was referenced Sep 23, 2026

test(think): cover a failed final persist on wait and recovery turns (#1997) #2342

Merged

Think reports a recovered turn as completed when final assistant message persistence fails #1997

Closed

Navigation Menu

Repository navigation

fix(think): preserve scheduled retries across restarts - #2297

fix(think): preserve scheduled retries across restarts#2297
ben-reitz merged 6 commits into
mainfrom
fix/2276-preserve-scheduled-chat-recovery-v2

ben-reitz commented Sep 17, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

pkg-pr-new Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

agent-think Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Sep 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ben-reitz commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Scope note

Uh oh!

changeset-bot Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

This comment was marked as resolved.

Uh oh!

pkg-pr-new Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agent-think Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 agents import sizes

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ben-reitz commented Sep 17, 2026 •

edited

Loading

changeset-bot Bot commented Sep 17, 2026 •

edited

Loading

pkg-pr-new Bot commented Sep 17, 2026 •

edited

Loading

agent-think Bot commented Sep 17, 2026 •

edited

Loading