Skip to content

fix(think): preserve scheduled retries across restarts - #2289

Closed
ben-reitz wants to merge 4 commits into
mainfrom
fix/2276-preserve-scheduled-chat-recovery
Closed

fix(think): preserve scheduled retries across restarts#2289
ben-reitz wants to merge 4 commits into
mainfrom
fix/2276-preserve-scheduled-chat-recovery

Conversation

@ben-reitz

@ben-reitz ben-reitz commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

After a restart, Think could mark a submission as failed even though its recovery was still in progress — or had already succeeded. Review found the reported bug plus adjacent gaps, all breaking the same invariant: a running submission must always be resolvable to live or terminal evidence at startup.

  1. Startup checked for scheduled continuations but overlooked scheduled retries (the reported bug).
  2. Recovery handed off to its replacement turn before that turn was durably recorded, and the replacement ran under a new request ID that startup couldn't match.
  3. A finished turn's recovery task could disappear before the submission ledger was updated — and any newly starting turn reclaimed the terminal stream that served as completion evidence.
  4. Facet-hosted recoveries never signaled handoff, holding the recovery step for the full model turn against its 15-minute timeout.
  5. The recovery lookup matched submission and request identities without precedence, so a collision could touch the wrong submission.

Fix

Recognize both kinds of recovery work at startup; keep the submission bound to whichever side currently owns it across the handoff (the handoff signal is scoped to the intended successor's call chain, on both the Tasks and facet-fiber paths); and settle the ledger from the turn's retained terminal stream without rerunning the turn or repeating callbacks.

ResumableStream gains a small additive lifecycle for that evidence: rows can be retained at settlement, are skipped by reclamation, and become reclaimable again on release(). Think pins a running submission's terminal stream until its ledger settles and releases pins at every settlement path and on startup. Exact submission IDs take precedence over request-ID fallback, and new recovery payloads carry the stable submission identity (old payloads still match).

 on restart
- only a scheduled continuation keeps the submission running
+ scheduled retries, the accepted replacement turn, and a finished
+ turn's retained terminal stream all keep or settle the submission correctly

Scope note

This fixes more than the original startup-detection gap: closing only item 1 still failed deterministic interruption tests for the reported symptom (see the reporter's third scenario in #2276, whose request-ID analysis matches item 2 exactly). The commits land on clean boundaries — f620ff54 (item 1), 2984fba5 + dfa76d65 (item 2), 91ee77ba (items 3–5 + the agents retention API, changeset: minor) — and can be split on request.

Validation: Deterministic red-first tests for every window above (including foreign-turn interference, foreign-stream reclamation with a real process restart, facet handoff, and ID collisions), full Think (936) and agents (3,244) suites, submission-recovery e2e, build, and pnpm run check — all passing with retries disabled.

Fixes #2276

@ben-reitz
fix(think): preserve scheduled recovery retries across restarts
f620ff5 
Startup treated only continuation callbacks as outstanding recovery, so an already-scheduled retry could lose its running submission after restart. Recognize both callbacks at the existing submission-recovery seam for Tasks and legacy schedules.

Cover pending retries, request isolation, terminal Tasks, cancellation, empty opened streams, and persisted restart completion. Refs #2276.
@changeset-bot

changeset-bot Bot commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 91ee77b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
agents Minor
@cloudflare/think Patch
@cloudflare/agent-think Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration[bot]
devin-ai-integration Bot reviewed Sep 16, 2026
View reviewed changes

@devin-ai-integration devin-ai-integration Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@agent-think

agent-think Bot commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟡 agents import sizes

Measured 343 runtime imports as minified bundles. The primary size is gzip; raw minified size is included for diagnosis. An existing import growing by more than 10% is marked red. This report is informational.

Red Yellow Green Unchanged New Removed
0 4 4 335 0 0

Compared 7e1b9fc1 with 01c321b4. Open workflow run.

Changed imports (8)
Status Import Base gzip Head gzip Delta
🟡 agents/chat#ResumableStream 5.1 KiB 5.2 KiB +175 B (+3.37%)
🟡 agents/streams#Streams 4.3 KiB 4.3 KiB +38 B (+0.87%)
🟡 agents/chat#createChatStreams 6.4 KiB 6.4 KiB +37 B (+0.57%)
🟡 agents/chat#createChatRecoveryTaskDefinition 2.6 KiB 2.6 KiB +1 B (+0.04%)
🟢 agents/chat#bumpChatRecoveryProgress 2.4 KiB 2.4 KiB -1 B (-0.04%)
🟢 agents/chat#sendIfOpen 2.4 KiB 2.4 KiB -1 B (-0.04%)
🟢 agents/chat#crossMessageToolResultUpdate 2.4 KiB 2.4 KiB -1 B (-0.04%)
🟢 agents/chat#applyAgentToolEvent 3.2 KiB 3.2 KiB -1 B (-0.03%)
All 343 current runtime imports
Status Import Gzip Raw minified
⚪ agents#__DO_NOT_USE_WILL_BREAK__agentContext 261.1 KiB 1141.0 KiB
⚪ agents#__DO_NOT_USE_WILL_BREAK__withInvocationScope 261.1 KiB 1141.0 KiB
⚪ agents#Agent 261.1 KiB 1141.0 KiB
⚪ agents#AGENT_TOOL_MILESTONE_PART 261.1 KiB 1141.0 KiB
⚪ agents#AGENT_TOOL_PROGRESS_PART 261.1 KiB 1141.0 KiB
⚪ agents#buildAgentPath 261.7 KiB 1143.3 KiB
⚪ agents#buildAgentUrl 261.8 KiB 1143.7 KiB
⚪ agents#callable 261.1 KiB 1141.0 KiB
⚪ agents#camelCaseToKebabCase 261.1 KiB 1141.0 KiB
⚪ agents#createHeaderBasedEmailResolver 261.3 KiB 1141.4 KiB
⚪ agents#DEFAULT_AGENT_STATIC_OPTIONS 261.1 KiB 1141.0 KiB
⚪ agents#DurableObjectOAuthClientProvider 261.1 KiB 1141.0 KiB
⚪ agents#getAgentByName 261.1 KiB 1141.0 KiB
⚪ agents#getCurrentAgent 261.1 KiB 1141.0 KiB
⚪ agents#getSubAgentByName 261.4 KiB 1141.6 KiB
⚪ agents#isDurableObjectCodeUpdateReset 261.1 KiB 1141.0 KiB
⚪ agents#isDurableObjectMemoryLimitReset 261.1 KiB 1141.0 KiB
⚪ agents#isDurableObjectStorageReset 261.1 KiB 1141.0 KiB
⚪ agents#isPlatformTransientError 261.1 KiB 1141.0 KiB
⚪ agents#MCP_SERVER_ID_MAX_LENGTH 261.1 KiB 1141.0 KiB
⚪ agents#MessageType 261.2 KiB 1141.3 KiB
⚪ agents#normalizeServerId 261.1 KiB 1141.0 KiB
⚪ agents#parseSubAgentPath 261.1 KiB 1141.0 KiB
⚪ agents#routeAgentEmail 261.4 KiB 1141.7 KiB
⚪ agents#routeAgentRequest 261.7 KiB 1142.9 KiB
⚪ agents#routeSubAgentRequest 261.3 KiB 1141.5 KiB
⚪ agents#SqlError 261.1 KiB 1141.0 KiB
⚪ agents#StreamingResponse 261.1 KiB 1141.0 KiB
⚪ agents#SUB_PREFIX 261.1 KiB 1141.0 KiB
⚪ agents#unstable_callable 261.2 KiB 1141.2 KiB
⚪ agents/agent-tools#agentTool 112.5 KiB 538.2 KiB
⚪ agents/browser#BrowserConnector 50.5 KiB 176.6 KiB
⚪ agents/browser#browserContent 36.3 KiB 127.4 KiB
⚪ agents/browser#browserExtract 36.3 KiB 127.4 KiB
⚪ agents/browser#browserLinks 36.3 KiB 127.4 KiB
⚪ agents/browser#browserMarkdown 36.3 KiB 127.4 KiB
⚪ agents/browser#browserPdf 36.3 KiB 127.3 KiB
⚪ agents/browser#BrowserRenderingError 36.0 KiB 126.7 KiB
⚪ agents/browser#browserScrape 36.3 KiB 127.4 KiB
⚪ agents/browser#browserScreenshot 36.3 KiB 127.3 KiB
⚪ agents/browser#browserSnapshot 36.3 KiB 127.4 KiB
⚪ agents/browser#CdpSession 37.2 KiB 129.8 KiB
⚪ agents/browser#CodemodeRuntime 39.6 KiB 139.0 KiB
⚪ agents/browser#connectBrowser 37.8 KiB 131.4 KiB
⚪ agents/browser#connectBrowserSession 37.5 KiB 130.4 KiB
⚪ agents/browser#connectUrl 37.6 KiB 130.5 KiB
⚪ agents/browser#createBrowserSession 36.3 KiB 127.5 KiB
⚪ agents/browser#DEFAULT_EXEC_SWEEP_IDLE_MS 36.0 KiB 126.6 KiB
⚪ agents/browser#DEFAULT_SWEEP_IDLE_MS 36.0 KiB 126.6 KiB
⚪ agents/browser#deleteBrowserSession 36.1 KiB 126.9 KiB
⚪ agents/browser#DurableBrowserSessionStore 36.4 KiB 127.6 KiB
⚪ agents/browser#getBrowserRecording 36.2 KiB 127.1 KiB
⚪ agents/browser#listBrowserTargets 36.1 KiB 126.9 KiB
⚪ agents/browser#loadCdpSpec 36.6 KiB 128.3 KiB
⚪ agents/browser#runQuickAction 36.0 KiB 126.6 KiB
⚪ agents/browser/ai#createBrowserRuntime 147.0 KiB 632.8 KiB
⚪ agents/browser/ai#createBrowserTools 147.0 KiB 632.9 KiB
⚪ agents/browser/ai#createQuickActionTools 122.5 KiB 554.3 KiB
⚪ agents/browser/tanstack-ai#createBrowserTools 162.7 KiB 701.7 KiB
⚪ agents/channels#ChannelHost 3.3 KiB 9.1 KiB
⚪ agents/channels#consumeChunks 237 B 321 B
⚪ agents/channels#createUserIdentityStore 1.3 KiB 3.2 KiB
⚪ agents/channels#fallback 155 B 160 B
⚪ agents/channels#fallbackChannel 893 B 1.9 KiB
⚪ agents/channels#fanout 151 B 156 B
⚪ agents/channels#fanoutChannel 875 B 1.9 KiB
⚪ agents/channels#identityKey 178 B 223 B
⚪ agents/channels#isChannelMessageSurface 271 B 452 B
⚪ agents/channels#linkChannelIdentities 113 B 99 B
⚪ agents/channels#matchesPath 110 B 96 B
⚪ agents/channels#routes 307 B 507 B
⚪ agents/channels#UserIdentityConflictError 248 B 335 B
⚪ agents/channels/ai-sdk#createSendMessageTool 112.2 KiB 537.4 KiB
⚪ agents/channels/ai-sdk#toChannelChunks 112.5 KiB 538.4 KiB
⚪ agents/channels/email#email 23.5 KiB 75.4 KiB
⚪ agents/channels/email#inboundEmail 22.3 KiB 72.3 KiB
⚪ agents/channels/slack#slack 5.5 KiB 14.6 KiB
⚪ agents/channels/slack#slackWebhook 2.2 KiB 5.1 KiB
⚪ agents/channels/tanstack-ai#createSendMessageTool 16.2 KiB 68.4 KiB
⚪ agents/channels/telegram#telegram 3.8 KiB 10.3 KiB
⚪ agents/channels/telegram#telegramWebhook 1.6 KiB 3.6 KiB
⚪ agents/channels/voice#browserVoice 1010 B 2.2 KiB
⚪ agents/chat#AbortRegistry 2.5 KiB 8.9 KiB
⚪ agents/chat#AGENT_TOOL_STREAM_PROGRESS_BUMP_THROTTLE_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#AgentToolProgressEmitter 2.7 KiB 9.5 KiB
⚪ agents/chat#AgentToolStreamProgressThrottle 2.4 KiB 8.3 KiB
⚪ agents/chat#aiSdkRecoveryCodec 2.3 KiB 8.2 KiB
🟢 agents/chat#applyAgentToolEvent 3.2 KiB 11.0 KiB
⚪ agents/chat#applyChunkToParts 2.3 KiB 8.2 KiB
⚪ agents/chat#applyToolUpdate 2.4 KiB 8.4 KiB
⚪ agents/chat#AutoContinuationController 2.3 KiB 8.2 KiB
⚪ agents/chat#awaitWithDeadline 2.4 KiB 8.4 KiB
⚪ agents/chat#broadcastTransition 3.2 KiB 11.4 KiB
⚪ agents/chat#buildChatRecoveringFrame 2.4 KiB 8.4 KiB
⚪ agents/chat#buildInClauseStrings 2.4 KiB 8.4 KiB
🟢 agents/chat#bumpChatRecoveryProgress 2.4 KiB 8.3 KiB
⚪ agents/chat#byteLength 2.5 KiB 8.5 KiB
⚪ agents/chat#CHAT_LAST_TERMINAL_KEY 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_MESSAGE_TYPES 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERING_FLAG_TTL_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERING_KEY 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_ALARM_DEBOUNCE_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_INCIDENT_KEY_PREFIX 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_INCIDENT_TTL_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_PROGRESS_KEY 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_STABLE_RETRY_DELAY_SECONDS 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_RECOVERY_TASK_NAME 2.3 KiB 8.2 KiB
⚪ agents/chat#CHAT_STREAM_PROGRESS_CREDIT_THROTTLE_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#ChatRecoveryEngine 4.4 KiB 15.3 KiB
⚪ agents/chat#chatRecoveryTaskRunOptions 2.4 KiB 8.6 KiB
⚪ agents/chat#ChatStreamStalledError 2.4 KiB 8.3 KiB
⚪ agents/chat#classifyAgentToolChildRecovery 2.4 KiB 8.5 KiB
⚪ agents/chat#clearChatTerminal 2.3 KiB 8.3 KiB
⚪ agents/chat#clientResolvableToolNames 2.3 KiB 8.3 KiB
⚪ agents/chat#ContinuationState 2.7 KiB 9.8 KiB
⚪ agents/chat#createAgentToolEventState 2.3 KiB 8.3 KiB
⚪ agents/chat#createChatFiberSnapshot 2.5 KiB 8.6 KiB
🟡 agents/chat#createChatRecoveryTaskDefinition 2.6 KiB 9.0 KiB
🟡 agents/chat#createChatStreams 6.4 KiB 22.2 KiB
⚪ agents/chat#createChatTurnTaskDefinition 2.7 KiB 8.9 KiB
⚪ agents/chat#createToolsFromClientSchemas 114.3 KiB 545.4 KiB
🟢 agents/chat#crossMessageToolResultUpdate 2.4 KiB 8.6 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_MAX_ATTEMPTS 2.3 KiB 8.2 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_MAX_OOM_RETRIES 2.3 KiB 8.2 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_MAX_WORK 2.3 KiB 8.2 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_NO_PROGRESS_TIMEOUT_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_STABLE_TIMEOUT_MS 2.3 KiB 8.2 KiB
⚪ agents/chat#DEFAULT_CHAT_RECOVERY_TERMINAL_MESSAGE 2.4 KiB 8.3 KiB
⚪ agents/chat#dispatchChatRecoveryToHandoff 3.0 KiB 9.9 KiB
⚪ agents/chat#drainInteractionApplies 2.3 KiB 8.3 KiB
⚪ agents/chat#enforceRowSizeLimit 3.5 KiB 11.2 KiB
⚪ agents/chat#hasIncompleteToolBatch 2.4 KiB 8.6 KiB
⚪ agents/chat#interceptAgentToolBroadcast 2.5 KiB 8.7 KiB
⚪ agents/chat#isPlatformFailure 2.6 KiB 8.9 KiB
⚪ agents/chat#isReplayChunk 2.4 KiB 8.7 KiB
⚪ agents/chat#iterateWithStallWatchdog 2.6 KiB 8.8 KiB
⚪ agents/chat#KV_DELETE_MAX_KEYS 2.3 KiB 8.2 KiB
⚪ agents/chat#listActiveChatRecoveryIncidents 2.4 KiB 8.4 KiB
⚪ agents/chat#MAX_BOUND_PARAMS 2.3 KiB 8.2 KiB
⚪ agents/chat#MessageType 2.4 KiB 9.0 KiB
⚪ agents/chat#normalizeToolInput 2.3 KiB 8.2 KiB
⚪ agents/chat#parseProtocolMessage 2.5 KiB 9.0 KiB
⚪ agents/chat#partAwaitsClientInteraction 2.4 KiB 8.6 KiB
⚪ agents/chat#pausedExecutionUpdate 2.4 KiB 8.4 KiB
⚪ agents/chat#pendingChatTerminal 2.3 KiB 8.3 KiB
⚪ agents/chat#persistReconstructedOrphan 3.1 KiB 11.0 KiB
⚪ agents/chat#PreStreamTurns 2.6 KiB 9.3 KiB
⚪ agents/chat#readChatRecoveryProgress 2.3 KiB 8.3 KiB
⚪ agents/chat#reconcileMessages 2.8 KiB 9.6 KiB
⚪ agents/chat#reconcileOrphanPartial 2.4 KiB 8.5 KiB
⚪ agents/chat#recordChatTerminal 2.4 KiB 8.3 KiB
⚪ agents/chat#repairInterruptedToolParts 2.6 KiB 9.1 KiB
⚪ agents/chat#resolveChatRecoveryConfig 2.6 KiB 8.9 KiB
⚪ agents/chat#resolveToolMergeId 2.4 KiB 8.5 KiB
🟡 agents/chat#ResumableStream 5.2 KiB 17.4 KiB
⚪ agents/chat#ResumeHandshake 3.0 KiB 10.4 KiB
⚪ agents/chat#ROW_MAX_BYTES 2.3 KiB 8.2 KiB
⚪ agents/chat#runChatRecoveryExhaustion 2.6 KiB 8.9 KiB
⚪ agents/chat#sanitizeMessage 2.5 KiB 8.8 KiB
🟢 agents/chat#sendIfOpen 2.4 KiB 8.4 KiB
⚪ agents/chat#setChatRecovering 2.5 KiB 8.5 KiB
⚪ agents/chat#shouldCreditStreamProgress 2.4 KiB 8.3 KiB
⚪ agents/chat#STREAM_RESUME_NONE_REASONS 2.3 KiB 8.2 KiB
⚪ agents/chat#StreamAccumulator 2.9 KiB 10.7 KiB
⚪ agents/chat#StreamProgressCreditThrottle 2.4 KiB 8.3 KiB
⚪ agents/chat#SubmitConcurrencyController 2.9 KiB 10.3 KiB
⚪ agents/chat#sweepStaleChatRecoveryIncidents 2.4 KiB 8.5 KiB
⚪ agents/chat#TextSegmentJoiner 2.7 KiB 9.2 KiB
⚪ agents/chat#TIMED_OUT 2.3 KiB 8.2 KiB
⚪ agents/chat#toolApprovalUpdate 2.4 KiB 8.5 KiB
⚪ agents/chat#toolPartHasSettledResult 2.3 KiB 8.4 KiB
⚪ agents/chat#toolResultUpdate 2.4 KiB 8.5 KiB
⚪ agents/chat#truncateOlderMessages 3.2 KiB 10.4 KiB
⚪ agents/chat#TurnQueue 2.6 KiB 9.2 KiB
⚪ agents/chat#unwrapChatFiberSnapshot 2.4 KiB 8.5 KiB
⚪ agents/chat#wrapChatFiberSnapshot 2.3 KiB 8.2 KiB
⚪ agents/chat-sdk#ChatSdkStateAdapter 263.5 KiB 1152.6 KiB
⚪ agents/chat-sdk#ChatSdkStateAgent 262.8 KiB 1150.0 KiB
⚪ agents/chat-sdk#createChatSdkState 263.5 KiB 1152.6 KiB
⚪ agents/chat-sdk#defaultKeyShard 261.2 KiB 1141.2 KiB
⚪ agents/chat-sdk#defaultThreadShard 261.1 KiB 1141.0 KiB
⚪ agents/chat/react#detectToolsRequiringConfirmation 3.3 KiB 8.3 KiB
⚪ agents/chat/react#extractClientToolSchemas 3.2 KiB 8.3 KiB
⚪ agents/chat/react#getAgentMessages 3.4 KiB 8.6 KiB
⚪ agents/chat/react#getToolApproval 3.1 KiB 8.0 KiB
⚪ agents/chat/react#getToolCallId 3.1 KiB 8.0 KiB
⚪ agents/chat/react#getToolInput 3.1 KiB 8.0 KiB
⚪ agents/chat/react#getToolOutput 3.1 KiB 8.0 KiB
⚪ agents/chat/react#getToolPartState 3.2 KiB 8.2 KiB
⚪ agents/chat/react#useAgentChat 132.9 KiB 609.7 KiB
⚪ agents/chat/react#WebSocketChatTransport 5.7 KiB 17.1 KiB
⚪ agents/chat/transport#WebSocketChatTransport 2.8 KiB 9.2 KiB
⚪ agents/client#AgentClient 21.8 KiB 75.2 KiB
⚪ agents/client#AgentConnectionError 16.3 KiB 57.8 KiB
⚪ agents/client#agentFetch 19.7 KiB 69.0 KiB
⚪ agents/client#createStubProxy 16.4 KiB 57.8 KiB
⚪ agents/client#DEFAULT_CALL_TIMEOUT_MS 16.2 KiB 57.6 KiB
⚪ agents/client#isTerminalCloseEvent 16.3 KiB 57.6 KiB
⚪ agents/client#nativeCall 16.5 KiB 58.3 KiB
⚪ agents/client#NativeCallQueue 16.7 KiB 58.9 KiB
⚪ agents/client#splitCallOptions 16.3 KiB 57.7 KiB
⚪ agents/context#AgentContextProvider 412 B 792 B
⚪ agents/context#AgentSearchProvider 640 B 1.3 KiB
⚪ agents/context#ContextBlocks 87.4 KiB 429.7 KiB
⚪ agents/email#createAddressBasedEmailResolver 193 B 227 B
⚪ agents/email#createCatchAllEmailResolver 110 B 97 B
⚪ agents/email#createHeaderBasedEmailResolver 334 B 492 B
⚪ agents/email#createSecureReplyEmailResolver 718 B 1.3 KiB
⚪ agents/email#DEFAULT_MAX_AGE_SECONDS 56 B 39 B
⚪ agents/email#isAutoReplyEmail 201 B 249 B
⚪ agents/email#signAgentHeaders 424 B 812 B
⚪ agents/experimental/webmcp#registerWebMcp 85.2 KiB 295.8 KiB
⚪ agents/lifecycle#getCurrentAgent 376 B 798 B
⚪ agents/lifecycle#Lifecycle 8.4 KiB 26.1 KiB
⚪ agents/lifecycle#LifecycleCapability 497 B 999 B
⚪ agents/mcp#createLegacyMcpHandler 378.6 KiB 1583.3 KiB
⚪ agents/mcp#createMcpHandler 390.9 KiB 1628.5 KiB
⚪ agents/mcp#DurableObjectEventStore 345.1 KiB 1441.8 KiB
⚪ agents/mcp#ElicitRequestSchema 345.1 KiB 1441.8 KiB
⚪ agents/mcp#experimental_createMcpHandler 378.7 KiB 1583.6 KiB
⚪ agents/mcp#getMcpAuthContext 345.1 KiB 1441.9 KiB
⚪ agents/mcp#MCP_SERVER_ID_MAX_LENGTH 345.1 KiB 1441.9 KiB
⚪ agents/mcp#McpAgent 345.1 KiB 1441.8 KiB
⚪ agents/mcp#normalizeServerId 345.1 KiB 1441.8 KiB
⚪ agents/mcp#RPC_DO_PREFIX 345.1 KiB 1441.8 KiB
⚪ agents/mcp#RPCClientTransport 345.1 KiB 1441.8 KiB
⚪ agents/mcp#RPCServerTransport 345.1 KiB 1441.8 KiB
⚪ agents/mcp#SSEEdgeClientTransport 345.2 KiB 1442.1 KiB
⚪ agents/mcp#StreamableHTTPEdgeClientTransport 345.2 KiB 1442.1 KiB
⚪ agents/mcp#WorkerTransport 348.5 KiB 1458.7 KiB
⚪ agents/mcp/client#getNamespacedData 62.9 KiB 240.0 KiB
⚪ agents/mcp/client#MCP_SERVER_ID_MAX_LENGTH 62.9 KiB 239.9 KiB
⚪ agents/mcp/client#MCPClientManager 158.6 KiB 702.6 KiB
⚪ agents/mcp/client#normalizeServerId 63.0 KiB 240.2 KiB
⚪ agents/mcp/do-oauth-client-provider#DurableObjectOAuthClientProvider 2.1 KiB 6.6 KiB
⚪ agents/mcp/server#createMcpHandler 80.5 KiB 307.2 KiB
⚪ agents/mcp/server#getMcpAuthContext 64.0 KiB 245.5 KiB
⚪ agents/observability#channels 259 B 549 B
⚪ agents/observability#genericObservability 470 B 1.2 KiB
⚪ agents/observability#subscribe 324 B 668 B
⚪ agents/observability/ai#wrapAISDK 8.8 KiB 30.5 KiB
⚪ agents/queue#Queue 3.1 KiB 9.6 KiB
⚪ agents/react#_testUtils 19.3 KiB 66.4 KiB
⚪ agents/react#useAgent 27.1 KiB 89.9 KiB
⚪ agents/react#useAgentToolEvents 21.1 KiB 73.7 KiB
⚪ agents/routing#getAgentByName 795 B 1.7 KiB
⚪ agents/routing#routeAgentRequest 1.6 KiB 3.6 KiB
⚪ agents/routing#RoutedAgents 2.4 KiB 6.2 KiB
⚪ agents/schedule#getSchedulePrompt 85.8 KiB 424.7 KiB
⚪ agents/schedule#scheduleSchema 85.3 KiB 423.6 KiB
⚪ agents/schedule#unstable_getSchedulePrompt 85.9 KiB 424.9 KiB
⚪ agents/schedule#unstable_scheduleSchema 85.3 KiB 423.6 KiB
⚪ agents/schedules#Scheduler 6.8 KiB 22.0 KiB
⚪ agents/schedules/parser#getSchedulePrompt 85.8 KiB 424.7 KiB
⚪ agents/schedules/parser#scheduleSchema 85.3 KiB 423.6 KiB
⚪ agents/sessions#COMPACTION_PREFIX 147 B 160 B
⚪ agents/sessions#createCompactFunction 1.7 KiB 4.0 KiB
⚪ agents/sessions#isCompactionMessage 177 B 200 B
⚪ agents/sessions#Session 2.0 KiB 6.3 KiB
⚪ agents/sessions#Sessions 9.6 KiB 34.3 KiB
⚪ agents/skills#fromManifest 309.8 KiB 1084.0 KiB
⚪ agents/skills#parseSkillFrontmatter 328.4 KiB 1146.2 KiB
⚪ agents/skills#parseSkillMarkdown 328.6 KiB 1146.5 KiB
⚪ agents/skills#r2 330.2 KiB 1150.4 KiB
⚪ agents/skills#runner 369.0 KiB 1297.8 KiB
⚪ agents/skills#SkillRegistry 416.4 KiB 1581.7 KiB
⚪ agents/skills/compile#compileSkillScript 15.4 KiB 43.4 KiB
⚪ agents/skills/compile#isCompilableSkillScript 15.4 KiB 43.3 KiB
⚪ agents/state#State 1.1 KiB 2.4 KiB
⚪ agents/streams#DEFAULT_MAX_CHUNK_BYTES 94 B 96 B
⚪ agents/streams#sseResponse 857 B 1.6 KiB
⚪ agents/streams#StreamClosedError 175 B 212 B
⚪ agents/streams#StreamNotFoundError 215 B 276 B
🟡 agents/streams#Streams 4.3 KiB 14.0 KiB
⚪ agents/streams#StreamSerializationError 171 B 201 B
⚪ agents/tasks#DuplicateTaskStepError 328 B 463 B
⚪ agents/tasks#MAX_SERIALIZED_BYTES 190 B 232 B
⚪ agents/tasks#MissingTaskDefinitionError 358 B 536 B
⚪ agents/tasks#NonRetryableError 238 B 308 B
⚪ agents/tasks#TaskReplayDivergedError 341 B 483 B
⚪ agents/tasks#Tasks 8.9 KiB 31.8 KiB
⚪ agents/tasks#TaskSerializationError 258 B 339 B
⚪ agents/types#MessageType 211 B 365 B
⚪ agents/vite#default 353.8 KiB 1356.1 KiB
⚪ agents/voice#addSFUTracks 324 B 424 B
⚪ agents/voice#createSFUSession 255 B 306 B
⚪ agents/voice#createSFUWebSocketAdapter 331 B 429 B
⚪ agents/voice#decodeVarint 157 B 160 B
⚪ agents/voice#downsample48kStereoTo16kMono 238 B 324 B
⚪ agents/voice#encodePayloadToProtobuf 189 B 279 B
⚪ agents/voice#encodeVarint 129 B 122 B
⚪ agents/voice#extractPayloadFromProtobuf 271 B 425 B
⚪ agents/voice#iterateText 1.6 KiB 3.8 KiB
⚪ agents/voice#renegotiateSFUSession 329 B 428 B
⚪ agents/voice#SentenceChunker 550 B 1.1 KiB
⚪ agents/voice#sfuFetch 294 B 358 B
⚪ agents/voice#upsample16kMonoTo48kStereo 211 B 272 B
⚪ agents/voice#VOICE_PROTOCOL_VERSION 51 B 31 B
⚪ agents/voice#withVoice 9.5 KiB 32.7 KiB
⚪ agents/voice#withVoiceInput 4.8 KiB 16.0 KiB
⚪ agents/voice#WorkersAIFluxSTT 1.5 KiB 4.3 KiB
⚪ agents/voice#WorkersAINova3STT 1.6 KiB 4.3 KiB
⚪ agents/voice#WorkersAITTS 682 B 1.4 KiB
⚪ agents/voice/client#VOICE_PROTOCOL_VERSION 473 B 768 B
⚪ agents/voice/client#VoiceClient 10.1 KiB 31.6 KiB
⚪ agents/voice/client#WebSocketVoiceTransport 4.6 KiB 13.3 KiB
⚪ agents/voice/errors#logVoiceError 132 B 173 B
⚪ agents/voice/errors#toVoiceError 94 B 80 B
⚪ agents/voice/errors#voiceErrorMessage 125 B 111 B
⚪ agents/voice/errors#VoiceProviderError 190 B 345 B
⚪ agents/voice/react#useVoiceAgent 13.6 KiB 42.5 KiB
⚪ agents/voice/react#useVoiceInput 13.3 KiB 41.4 KiB
⚪ agents/voice/react#WebSocketVoiceTransport 7.4 KiB 21.3 KiB
⚪ agents/voice/sfu#addSFUTracks 323 B 424 B
⚪ agents/voice/sfu#createSFUSession 254 B 306 B
⚪ agents/voice/sfu#createSFUWebSocketAdapter 329 B 429 B
⚪ agents/voice/sfu#decodeVarint 155 B 160 B
⚪ agents/voice/sfu#downsample48kStereoTo16kMono 236 B 324 B
⚪ agents/voice/sfu#encodePayloadToProtobuf 188 B 279 B
⚪ agents/voice/sfu#encodeVarint 129 B 122 B
⚪ agents/voice/sfu#extractPayloadFromProtobuf 270 B 425 B
⚪ agents/voice/sfu#renegotiateSFUSession 329 B 428 B
⚪ agents/voice/sfu#sfuFetch 292 B 358 B
⚪ agents/voice/sfu#upsample16kMonoTo48kStereo 209 B 272 B
⚪ agents/voice/text#iterateText 1.6 KiB 3.8 KiB
⚪ agents/voice/text#SentenceChunker 549 B 1.1 KiB
⚪ agents/voice/types#VOICE_PROTOCOL_VERSION 51 B 31 B
⚪ agents/voice/workers-ai#WorkersAIFluxSTT 1.5 KiB 4.3 KiB
⚪ agents/voice/workers-ai#WorkersAINova3STT 1.6 KiB 4.3 KiB
⚪ agents/voice/workers-ai#WorkersAITTS 682 B 1.4 KiB
⚪ agents/websockets#CAPNWEB_TRANSPORT_QUERY 12.4 KiB 43.3 KiB
⚪ agents/websockets#CAPNWEB_TRANSPORT_VALUE 12.4 KiB 43.3 KiB
⚪ agents/websockets#capnWebTransportUrl 12.5 KiB 43.5 KiB
⚪ agents/websockets#CF_NO_PROTOCOL_KEY 12.4 KiB 43.3 KiB
⚪ agents/websockets#CF_READONLY_KEY 12.4 KiB 43.3 KiB
⚪ agents/websockets#isCapnWebTransportUpgrade 12.5 KiB 43.4 KiB
⚪ agents/websockets#registerInternalConnectionKeys 12.4 KiB 43.4 KiB
⚪ agents/websockets#WebSockets 20.0 KiB 69.3 KiB
⚪ agents/workflows#AgentWorkflow 262.4 KiB 1145.7 KiB
⚪ agents/workflows#WorkflowRejectedError 261.2 KiB 1141.2 KiB
⚪ agents/x402#normalizeNetwork 14.7 KiB 61.1 KiB
⚪ agents/x402#withX402 23.0 KiB 89.2 KiB
⚪ agents/x402#withX402Client 104.1 KiB 346.5 KiB

Reported by agent-think[bot].

@pkg-pr-new

pkg-pr-new Bot commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2289

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2289

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2289

hono-agents

npm i https://pkg.pr.new/hono-agents@2289

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2289

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2289

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2289

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2289

commit: 91ee77b

@ben-reitz
fix(think): keep submission ownership across recovery handoff
2984fba 
Review found two remaining interruption windows: the recovery callback signaled handoff before the successor turn was durably accepted, and a non-retained successor could settle before ledger completion. Rebind the running submission to the accepted successor before handoff, retain terminal stream evidence for running submissions, and settle the ledger from that evidence at startup without rerunning the turn.

Replace the e2e table-surgery staging with inert seeded evidence driven through two real restarts so production owns retry scheduling and linkage.
devin-ai-integration[bot]

This comment was marked as resolved.

Sign in to view

@ben-reitz
fix(think): scope recovery handoff signal to the successor turn
dfa76d6 
Review found the acceptance callback lived on the agent instance, so any concurrent root chat turn admitted during the successor's pre-admission awaits could claim the handoff, rebind the submission to a foreign request, and release the recovery Task early. Hold the callback in successor-scoped async context so only the intended turn's chain can signal acceptance.
devin-ai-integration[bot]
devin-ai-integration Bot reviewed Sep 17, 2026
View reviewed changes

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

Devin Review

Comment thread packages/think/src/think.ts Outdated
Comment on lines +5079 to +5086
const acceptance = recoveredTurnAcceptanceContext.getStore();
const onAccepted =
acceptance?.agent === this ? acceptance.onAccepted : undefined;
const run = (): Promise<T> => {
// Tasks accepts the run durably before invoking its live closure. Rebind
// submission ownership before telling the recovery Task it may settle,
// so startup can always find either predecessor or successor evidence.
onAccepted?.(requestId);

@devin-ai-integration devin-ai-integration Bot Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Facet recovery never hands off

When recovery runs on a facet, _runChatRecoveryFiber takes the legacy-fiber branch before onAccepted is captured. The recovery dispatcher then waits for the full model run inside a 15-minute recovery step. Long turns time out, while shorter turns block later lifecycle jobs.

Learn more

Facet-hosted chat turns use the legacy fiber engine because routed sub-agents cannot create chat Tasks. The new acceptance callback is read only after that branch returns, so a facet successor never calls the recovery dispatcher's handoff signal. The dispatcher therefore keeps the recovery Task or compatibility schedule attached until the complete model turn returns. The shared recovery Task bounds that callback with a 15-minute step timeout in createChatRecoveryTaskDefinition, despite chat turns supporting longer execution. It also keeps the Lifecycle alarm job occupied, delaying unrelated work on the same object.

Example: A routed Think facet recovers a provider turn that takes 20 minutes. The recovery callback never hands off, its continuation step times out after 15 minutes, and the platform can retry a turn that was still executing.

Recommended fix: Add an acceptance signal for the legacy-fiber branch at the point _runFiberWithStashWrapper has durably created the successor fiber. Preserve the same ordering as the Task branch: persist or rebind the submission first, then invoke onTurnStarted. Add facet recovery tests covering the handoff boundary and a queued lifecycle job.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread packages/think/src/think.ts Outdated
Comment on lines +13657 to +13664
if (this._agentToolRunsByRequestId.get(requestId)) return false;
this._ensureSubmissionTable();
const running = this.sql<{ submission_id: string }>`
SELECT submission_id FROM cf_think_submissions
WHERE request_id = ${requestId} AND status = 'running'
LIMIT 1
`;
return running.length === 0;

@devin-ai-integration devin-ai-integration Bot Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Next turn erases recovery evidence

If another turn starts before recovery updates the ledger, ResumableStream.start deletes the retained terminal stream. A crash then leaves no recovery evidence, so startup marks the completed submission as error.

Learn more

A recovered successor keeps its terminal stream because its non-retained chat Task can disappear before _completeRecoveredSubmission updates the submission row. After the queued turn finishes, recovery still awaits incident bookkeeping before that ledger update. Another admitted turn can start during this await. ResumableStream.start() calls reclaim(), which deletes every terminal chat stream without checking whether a running submission still depends on it. A restart after that deletion cannot find an active recovery Task, successor chat Task, or terminal stream.

Example: Recovery successor r2 finishes and retains a completed stream while submission s1 is still running. An unrelated turn r3 starts while incident completion is awaiting storage, deleting r2's stream. The object restarts before s1 is finalized, and startup changes s1 to error instead of completed.

Recommended fix: Prevent reclaim from deleting terminal streams referenced by running submission rows, or atomically settle the submission ledger before making the stream reclaimable. Add a test that pauses incident completion, starts an unrelated stream, then runs startup reconciliation.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread packages/think/src/think.ts Outdated
Comment on lines 15934 to 15936
WHERE (submission_id = ${recoveredRequestId} OR request_id = ${recoveredRequestId})
AND status = 'running'
LIMIT 1

@devin-ai-integration devin-ai-integration Bot Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Recovery IDs select wrong submission

When one submission ID equals another submission’s successor request ID, _readRunningSubmissionForRecovery can select either running row. request_id is non-unique in the submission schema. The callback can rebind or terminalize the wrong submission.

Learn more

Submission IDs are caller-controlled primary keys, while successor request IDs occupy a separate, non-unique column. Recovery payloads normally carry the stable submission ID, but the lookup now treats both namespaces as interchangeable. Two running rows can therefore match one value, and LIMIT 1 provides no deterministic ownership rule. The same identity ambiguity also affects scheduled-recovery matching during startup.

Example: Submission A is recovering under successor request ID r2. After observing r2, a caller creates submission ID r2; both remain running. Recovery for the new r2 submission can select A through its request_id and complete or rebind A instead.

Recommended fix: Preserve the identity kind in recovery payloads and query submission_id first when the payload carries the stable submission identity. If successor IDs must also be accepted for compatibility, use an explicit fallback only when no submission-ID match exists, and apply the same precedence in _hasScheduledChatRecovery.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@cloudflare cloudflare deleted a comment from devin-ai-integration Bot Sep 17, 2026
@ben-reitz

ben-reitz commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

/devin review

@devin-ai-integration

devin-ai-integration Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Starting Devin Review.

Devin Review

@ben-reitz
fix(think): close facet handoff, evidence reclamation, and identity-c…
91ee77b 
…ollision gaps

Review of the handoff work found three follow-ups: facet-hosted recoveries never signaled acceptance, so the recovery step awaited the whole model turn against its 15-minute timeout; stream reclamation deleted the terminal evidence retained for running submissions as soon as any new turn started; and the recovery lookup matched submission and request identities without precedence, letting a collision touch the wrong submission.

ResumableStream now owns a durable retained-row lifecycle (retain at settlement, skip at reclaim, release when settled) and Think releases pins at every ledger settlement and on startup. Facet successors signal acceptance once their fiber is durably created. Exact submission identity takes precedence over request-id fallback, and new recovery payloads carry the stable submission id.
@ben-reitz
ben-reitz force-pushed the fix/2276-preserve-scheduled-chat-recovery branch from 01c321b to 91ee77b Compare September 17, 2026 21:50
@ben-reitz

ben-reitz commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #2297 (identical content on a fresh branch) to work around a stuck automated review run on this PR.

@ben-reitz ben-reitz closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@devin-ai-integration devin-ai-integration[bot] devin-ai-integration[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Think recovery skips an interrupted opened stream with no assistant content

1 participant

@ben-reitz