Repository navigation
Conversation
🦋 Changeset detectedLatest commit: 3ee85b6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
@mmarufov is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Author
|
@vercel/workflow Ready for review. Could you authorize the preview deployments and rerun the blocked CI checks? Thanks! |
Signed-off-by: Muhammadjon Marufov <marufov254@gmail.com>
mmarufov
force-pushed
the
mmarufov/workflow-4836-run-recovery
branch
from
October 19, 2026 22:00
cb9a08f to
3ee85b6
Compare
2 of 3 tasks
NishitaSikka
added a commit
that referenced
this pull request
Oct 20, 2026
…ion (#4922) Closes #4836. Alternative to #4911. ## Problem Local World creates a run in two writes: the run record, then `run_created` in slot 1. If the process dies between them, a later `run_started` sees the run record, skips resilient start, and takes slot 1. The run becomes `running` with no `run_created`, which replay needs. ## Fix Before `run_started` or `attr_set` can become a pending run's first event, restore `run_created` from the saved run record into slot 1. `run_cancelled` and `run_failed` attempt the same repair, but log and continue if the history cannot be repaired because terminal transitions do not replay. Only the run record is trusted, never the redelivered event's data. Event-first creation was considered and rejected: it flips the gap. A `run_created` without a run record is invisible to recovery (which scans `runs/`), and a redelivery could write a duplicate. ## Edge cases - For `run_started` and `attr_set`, repair only succeeds for pending runs with an empty log and persisted input. Missing input or non-empty history without `run_created` throws a `WorkflowWorldError` and leaves the run untouched. - For `run_cancelled` and `run_failed`, the same repair is attempted, but failure is logged and the terminal transition proceeds so a corrupt pending run can still be stopped. - Slot 1 is written with the existing exclusive create (temp file, then hard link). If another writer won, its `run_created` is accepted; any other occupant is an error. `run_created` never moves to another slot. - A slow original writer that finishes after the repair converges on the existing creation event. A resilient-start writer that loses slot 1 re-reads the run so an already-running run takes the normal idempotent path. - Running runs don't read their log. Pending runs pay one extra read on their first lifecycle event. - A successful repair logs `restored missing run_created for <runId> (creation was interrupted or is still in flight)`. ## Crash safety No lock is added. The exclusive create holds nothing a crashed writer could leave behind: a crash before the hard link leaves slot 1 free, and a crash after it leaves a complete event. Existing cross-process lock files go stale after 30s. `allowReservedAttributes` is intentionally left off the restored event: it's only used when attributes are validated at creation, and nothing reads it from `run_created` during replay. ## Tests 17 recovery cases cover: - interrupted-creation repro, plus redelivery with different data (the run record wins) - 12 concurrent storage instances producing exactly one `run_created` - slow original writer and concurrent resilient-start writer convergence without duplicate creation events - missing input / non-empty log: explicit error and no mutation for replaying events - cancellation and failure still reaching terminal state when repair is impossible - `attr_set`, `run_cancelled`, and `run_failed` repair before publishing when repair is possible - no regressions: running runs read no event files; normal and resilient creation remain unchanged - crash states simulated directly on disk: no event temp file, a leftover temp file before the hard link, and a complete slot-1 event after the hard link; recovery completes without hanging or duplicating `run_created` With `main`'s `events-storage.ts`, 14 of the 17 cases fail. These tests cover the storage invariant. End-to-end replay coverage is left as a TODO in the test file, to be handled by a future test setup for data migrations / cross-version upgrades. `world-local` tests, typecheck, build, changeset check, and Biome pass (five pre-existing warnings in `events-storage.ts` after merging current `main`). --------- Signed-off-by: Nishita Sikka <nishita.sikka26@gmail.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Contributor
Author
|
Superseded by #4922. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #4836.
Local World saves the run record before publishing
run_created. If creation is interrupted between those writes, recovery treats the saved record as sufficient and appendsrun_startedalone, leaving replay without its creation history and original input.Restore the missing creation event from the persisted record before a pending run starts, updates attributes, or is cancelled or failed. The recovered event preserves the original input, workflow/deployment, context, attributes, encryption metadata, timestamp, and spec version.
Recovery and resilient creation share an exclusive first-slot write, so concurrent recoveries and a delayed original creator converge on one
run_created. Existing creation events are accepted anywhere in the log, and unrelated slot collisions retain normal retry behavior.Reconstruction requires empty history and saved input. Start and attribute updates reject unsafe reconstruction; cancellation and failure remain available for damaged runs. Running replays keep their existing path, with no recovery scan or additional lock.
How did you test your changes?
SIGKILLand restart probe completes in both Node and QuickJS with the original input and result.PR Checklist - Required to merge
@workflow/world-local.@vercel/workflowin a comment once the PR is ready and the checklist above is complete.