Skip to content

onChatRecovery is never invoked when turn is ended by stream-stall watchdog, leaving SSE stream open indefinitely #2042

Closed

Description

@bartrumb

Title

onChatRecovery is never invoked when turn is ended by stream-stall watchdog, leaving SSE stream open indefinitely

Description

Describe the bug
When a chat turn is terminated by the stream-stall watchdog, the recovery engine opens an incident, schedules _chatRecoveryContinue, and emits chat:recovery:skipped without ever calling the agent's onChatRecovery hook.

Additionally, the server-side chat() promise never settles, so neither onDone nor onError fires, leaving the client's SSE response (text/event-stream) open indefinitely after receiving the initial start frame (measured up to 180s before timing out).

Note: onChatRecovery works as expected on Durable Object eviction / deploy interruptions.

To Reproduce
Steps to reproduce the behavior:

  1. Configure a low stall threshold in beforeTurn:
    return { chatStreamStallTimeoutMs: 300 };
  2. POST a normal chat turn and read the SSE body using a bounded reader.
  3. Observe that only 1 frame (data: {"type":"start",...}) is emitted.
  4. The stream remains open indefinitely, onChatRecovery is never invoked, and wrangler tail logs show chat:recovery:skipped with attempt: 1, maxAttempts: 10, recoveryKind: "continue".

Expected behavior
Per the documented lifecycle in ChatRecoveryEngine.handleChatFiberRecovery (step 3 & 5):

  1. An interrupted turn with remaining budget should invoke onChatRecovery.
  2. Active streams should be closed/completed via completeRecoveredStream(streamId) to prevent client hangs.

Screenshots
N/A (See Wrangler tail JSON trace in additional context).

Version:

  • agents: 0.20.1 (also reproduced on 0.16.2)
  • @cloudflare/think: 0.15.1 (also reproduced on 0.9.1)

Additional context

Wrangler Tail JSON Output
{
  "wallTime": 271464,
  "cpuTime": 9,
  "executionModel": "durableObject",
  "outcome": "ok",
  "entrypoint": "ChatAgentThink",
  "diagnosticsChannelEvents": [
    {
      "channel": "agents:schedule",
      "message": {
        "type": "schedule:execute",
        "agent": "ChatAgentThink",
        "payload": { "callback": "_chatRecoveryContinue", "id": "5u-O5CfPM" }
      }
    },
    {
      "channel": "agents:chat",
      "message": {
        "type": "chat:recovery:skipped",
        "agent": "ChatAgentThink",
        "payload": {
          "incidentId": "b5cca0e0-fde5-4ca1-b1bf-18fae2646a7d:m_1785782647096",
          "requestId": "b5cca0e0-fde5-4ca1-b1bf-18fae2646a7d",
          "attempt": 1,
          "maxAttempts": 10,
          "recoveryKind": "continue"
        }
      }
    }
  ],
  "logs": []
}

Source Trace:
In agents/dist/chat/index.js, invokeOnChatRecovery is only reachable inside handleChatFiberRecovery. The stall path arrives via the scheduled _chatRecoveryContinue callback and reaches chat:recovery:skipped without entering handleChatFiberRecovery.

Environment details:

  • Runtime: Cloudflare Workers
  • Compatibility Date: 2026-02-01
  • Flags: nodejs_compat
  • Architecture: Turn delegated to DO from an Astro API route; DO returns SSE stream.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions