Skip to content

Postgres World - Workflows doesn't recover after process crash #1531

Closed

Description

@Tankilevitch

Description

Expanding on #679 — this isn't limited to sleep(). All workflow execution is non-recoverable after a process crash in Postgres World, including mid-step execution.

Root cause

As @jcourson-bg identified in #679, the postgres world delegates to localWorld.queue() which fires execution in an unawaited IIFE. The graphile-worker job completes immediately, so after a crash there's nothing left to retry — the event log is in postgres but nothing re-triggers the workflow.

Reproduction

Tested with workflow@4.2.0-beta.67, @workflow/world-postgres@4.1.0-beta.42, PostgreSQL 16, Next.js dev server.

Using the postgres example with WORKFLOW_TARGET_WORLD="@workflow/world-postgres".

Scenario 1: sleep() (confirms #679)

Workflow:

export async function handleUserSignup(email: string) {
  "use workflow";
  const user = await createUser(email);
  await sendWelcomeEmail(user);
  await sleep("2m");
  await sendOnboardingEmail(user);
  return { userId: user.id, status: "onboarded" };
}
  1. Trigger workflow — createUser and sendWelcomeEmail complete, enters sleep("2m")
  2. Verify: workflow_waits shows status = 'waiting', resume_at set 2 minutes out
  3. Kill the server (kill -9)
  4. Wait past resume_at
  5. Restart the server

Result: Run stays running, wait stays waiting (past resume_at), 0 graphile-worker jobs. sendOnboardingEmail never executes.

Scenario 2: Long-running step (no sleep())

Workflow:

export async function handleUserSignup(email: string) {
  "use workflow";
  const user = await createUser(email);
  await sendWelcomeEmail(user);
  await processUserData(user);
  await sendOnboardingEmail(user);
  return { userId: user.id, status: "onboarded" };
}

async function processUserData(user: { id: string; email: string }) {
  "use step";
  console.log(`Processing user data for: ${user.id}`);
  await new Promise((resolve) => setTimeout(resolve, 5 * 60 * 1000));
  return { processed: true };
}
  1. Trigger workflow — createUser and sendWelcomeEmail complete, processUserData starts
  2. Verify: events show step_started for processUserData
  3. Kill the server (kill -9) mid-step
  4. Restart the server

Result: Run stays running, step has step_started but no step_completed, 0 graphile-worker jobs. No recovery activity in server logs. The step is never retried.

DB state after restart (both scenarios)

workflow.workflow_runs:           status = 'running'
graphile_worker._private_jobs:   (0 rows)

Expected behavior

Both scenarios should recover after server restart. The postgres world's value proposition is durability - workflows should survive process crashes regardless of whether they're in a sleep() or mid-step execution.

Activity

  1. changed the title [-]Workflows don't recover after process crash in Postgres World — affects both `sleep()` and step execution[/-] [+]Postgres World Isn't durable - Workflows don't recover after process crash[/+] on Mar 27, 2026
  2. changed the title [-]Postgres World Isn't durable - Workflows don't recover after process crash[/-] [+]Postgres World - Workflows doesn't recover after process crash[/+] on Mar 27, 2026
  3. pranaygp commented on Jun 21, 2026

    @pranaygp
    Contributor

    Cross-link: #2544 addresses the root cause described here (no boot-time recovery for self-hosted worlds). It adds ensureWorldStarted() wired into framework startup, which boots the queue worker and re-enqueues active runs on restart, with an e2e test covering local + postgres (sleep → kill mid-flight → restart with no op → resumes).

    This issue is already closed by #1534 but #2544 adds a cleaner API and more documentation to setup run recovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions