Durable run streams lose acknowledged chunks, and the live read blocks forever at the hole #4306

Activity

1. rpinckne commented on Sep 23, 2026

   

   rpinckne

   on Sep 23, 2026

   Author

   More actions

   Another data point, this time on appends that get no response at all.

   Setup: @workflow/core 5.0.0-beta.55 and @workflow/world-vercel 5.0.0-beta.50, on the HTTP stream transport (not WebSocket). Each stream PUT has a wire deadline of 10 s plus 1 s per 64 KiB of body. The deadline is armed only once the request holds a connection, so pool wait is not counted.

   • Two appends of about 100 B each got no response headers. Both failed with UND_ERR_HEADERS_TIMEOUT, after 11,210 ms and 11,301 ms. They were on two different runs' _user streams, 35 s apart.
   • Neither chunk was stored. Reading both streams back afterwards, the chunk is missing.
   • Stream writes otherwise run at about 83 ms p50 and 384 ms p99. These stalls look like lost requests, not a slow tail.
   • The rate is about 1 in 7,000 to 10,000 writes, and hits come in bursts. Earlier releases show the same signature (core beta.53, 11,025 ms), so this is not a regression in the current betas.

   Why it matters: without a response, the client cannot tell whether the server kept the body. Until writer/sequence dedupe lands (#2731), an append that times out cannot be retried safely, because a late commit plus a resend duplicates bytes. The only safe choice today is to fail the step, and that loses the whole run.

   Can a request the client has already abandoned still commit on the server later? That answer decides whether dropping it and continuing is safe.
2. pranaygp commented on Sep 23, 2026

   

   pranaygp

   on Sep 23, 2026 – with Vercel

   Contributor

   More actions

   (AI)

   Severity: S1

   Acknowledged chunks go missing after they were durable and readable, so this is data loss. Any live reader that starts at or before a hole stalls: each session lasts the server's max response duration (~2 min), and the SDK reconnects to the same index up to 50 times before it errors, which is well over an hour per hole. The loss appears to scale with concurrent write load. It happens in the Vercel World's storage, outside this repo, so the SDK can't restore the data. What the SDK can fix is how a reader behaves at a hole.

   Reproduction

   • Chunk loss: I couldn't reproduce this locally. World Local and World Postgres assign chunk indices from storage order, so a hole can't form there. It only shows up under concurrent load on the Vercel World. Your run IDs and missing indices are the right evidence for the platform team.
   • Reader stall at a hole: reproduced against a world that serves the pattern you describe: chunks 0–1 are delivered, chunk 2 is missing, the tail is 5, and every read at index 2 ends without a frame. createReconnectingFramedStream reopened index 2 fifty times and then failed with a generic "exceeded maximum reconnection attempts". This is covered by the new tests in the PR below.
   • Streams never report closed: reproduced on World Local with a fresh run. A step writes 3 chunks and calls releaseLock() without close(), and the run completes. Afterwards streams.getInfo still reports done: false, and run.getReadable() delivers the 3 chunks and then waits indefinitely. Nothing in the SDK closes a run's streams when the run finishes, so the docs' "automatically closed when the workflow run completes" is currently inaccurate for every World. Until that changes, call close() on the writable after the last write, as a workaround.

   Related issues

   • Streams aren't closed automatically when runs end #964: the "streams never report closed" question is a duplicate of this. Please follow that issue for auto-closing streams on terminal run states.
   • Stream writes carry no idempotency key, so a retried step re-appends its chunks #3387 and [core] Ack'd write-channel stream writes with framed-v2 writer markers #2731: the ambiguous-append problem from your follow-up comment. Stream writes carry no idempotency key, so a PUT that times out can't be retried safely. [core] Ack'd write-channel stream writes with framed-v2 writer markers #2731 adds per-writer (writerId, seq) markers so readers can dedupe resent chunks. Whether a request the client abandoned can still commit later is a question for the maintainers.
   • Stream event delivery to clients lags 2-3 minutes behind durable writes (steps finish in seconds) #2767 (stream delivery lag on Vercel) and world-postgres: stream readers can stall after LISTEN disconnects or missed NOTIFY event #1855 (World Postgres readers stalling after a missed NOTIFY): similar stall symptoms with different causes.

   Fix

   #4334 changes the reader side and passes CI. When consecutive read sessions end without delivering a frame even though streams.getInfo reports a chunk at the resume index, the read now fails after 3 sessions instead of 50. The error is a StreamError that names the missing index and the tail, so you can resume with startIndex set to index + 1. The threshold can be tuned with WORKFLOW_FRAMED_STREAM_MAX_STALLS_BELOW_TAIL. Readers waiting at the tail for new data behave as before, and sessions that deliver data make no extra metadata requests. The reader deliberately doesn't skip a hole on its own, because silently dropping a chunk (such as a UI message start part) can corrupt consumer state. This doesn't restore lost chunks or fix the storage-side loss, so this issue stays open for that.

   To see if this fix solves your issue, could you try to install the pre-release in your package.json? Just replace the workflow dep like so:

   {
     "dependencies": {
       "workflow": "https://workflow-tarballs-bwxk2rih5.labs.vercel.dev/workflow.tgz",
       "@workflow/core": "https://workflow-tarballs-bwxk2rih5.labs.vercel.dev/workflow-core.tgz"
     }
   }
3. rpinckne commented on Sep 25, 2026

   

   rpinckne

   on Sep 25, 2026

   Author

   More actions

   New data on the storage-side loss: it happens in a window after a large concurrent launch, and its rate grows with launch width.

   Setup: @workflow/core 5.0.0-beta.55, @workflow/world-vercel 5.0.0-beta.50, HTTP stream transport. No writer saw a failed or unanswered append, and a live reader received the chunks before they went missing.

   We compared streams.getInfo tails against a full streams.getChunks listing for runs from three launches:

   launch	runs started together	chunks missing (long runs)
   2026-09-24, several small batches	3 to 14	0.02 to 0.31%
   2026-09-24, one launch	235	4.9 to 5.8%
   2026-09-25, one launch	285	13.3 to 14.7%

   For the 2026-09-25 launch, 170 of 321 streams have holes: 365,445 of 3,098,200 indices are missing (11.8%).

   • The loss is a time window. From about two minutes after launch until the burst eased (17:46 to 17:51 UTC on 9/24, 19:24 to 19:35 UTC on 9/25), the store dropped a fifth to a third of every writer's chunks. Text deltas, tool results, heartbeats and step markers were all affected. Chunks written after the window have no holes.
   • Chunks are dropped one at a time. Hole lengths fall off geometrically at the local loss rate, as independent per-chunk drops would. A lost multi-chunk PUT or a truncated tail would look different.
   • Examples (tail index, missing, separate holes, first missing):
     • wrun_41M3D0AA8J0GQXA9GK3ZJ3EVFK: 132373, 15,582, 9,049, 339
     • wrun_41M3D0A9QA0GQ1YAV2HFX6AHQG: 141124, 14,582, 6,118, 145
     • wrun_41M3D0ABKJ0GX5A6SJK70NTM4V: 101961, 13,334, 6,850, 0

   A related effect in the same window: appends slowed enough that one step whose model call had already finished hit core's drain timeout (Timed out draining step stream writes after 30000ms, step step_01M3D0A29D50EMEXMAK3NPPA7Y, 19:27 UTC). The step then retried and repeated the provider call.

   Questions for the platform team:

   1. Is there a per-project append rate or capacity limit behind this? If appends above it return 2xx and are dropped later, a 429 or other backpressure at write time would let writers slow down instead of losing data.
   2. Until it's fixed, is there a write pattern that avoids the window, such as fewer and larger appends or a cap on concurrent writers per project?
4. rpinckne commented on Sep 25, 2026

   

   rpinckne

   on Sep 25, 2026

   Author

   More actions

   A correction to my last comment, with stronger evidence: these chunks are lost after they were durable and readable, so this isn't write-time capacity. Please disregard my question about an append rate limit.

   For the 2026-09-25 launch, each run's stream was read twice: once live while the run ran, and once as a full replay from index 0 right after it finished.

   • 124 holed runs replayed complete between 19:25 and 19:49 UTC. Every text, reasoning and tool-input delta the live reader had received was present: 524,872 deltas in total. A streams.getChunks listing of the same runs at 22:23 UTC shows 127,689 of those indices missing.
     • Example: wrun_41M3D0A6090GXWCNJMM87WX7QQ replayed complete at 19:32:12 UTC. At 22:23 UTC it had tail index 29337 with 10,644 indices missing.
   • 46 runs replayed later (19:33 to 20:04 UTC) were already short at replay time (124,925 deltas), and shorter still by 22:23 (169,916).
   • Re-listing three runs at about 22:50 UTC gave the same counts as at 22:23. The loss happened in the hours after the runs, then stopped.

   So the chunks were acknowledged, delivered live, and readable in a full replay after the run, then removed within roughly 2.5 hours. That points to something after the write path (replication, compaction, retention or a failover), not to the appends. It also rules out the client: an append that was never acknowledged couldn't have been served in a complete replay.

   Questions:

   1. What happens to a stream's chunks between acknowledgement and long-term storage? Can anything remove or rewrite chunks after a run completes?
   2. Is the platform team investigating this as data loss? Happy to share more run ids and exact index lists.
5. pranaygp

   assigned

   alangenfeld

   on Sep 29, 2026
6. alangenfeld commented on Sep 29, 2026

   

   alangenfeld

   on Sep 29, 2026

   Collaborator

   More actions

   Hey @rpinckne thanks for the thorough detailed report. I'm sorry for this data loss you incurred and am working to prevent it from happening again.

   The run ids are sufficient if you want to report any new instances you observe.