world-postgres: crash recovery runs only at startup, so a stranded run waits for a deploy #3993

Packages: @workflow/world-postgres@4.3.0, @workflow/world@4.2.1
Runtime: AWS ECS Fargate, rolling deploys

What happens

reenqueueActiveRuns is the only mechanism that creates fresh work for a run whose job was lost, and it is called exactly once, inside start():

async start() {
  await queue.start();
  await reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres');
}

There is no periodic equivalent, and no documented way to invoke the sweep against an already-running world. So a run stranded by a lost job recovers only when some process boots.

Why this matters

On a rolling-deploy platform, "some process boots" means "someone deploys". In a busy environment that looks self-healing; in production, where releases are occasional, a stranded run waits for the next one.

Meanwhile the run stays running in workflow_runs, so anything reading run status believes work is in progress.

Worth noting separately: the sweep reports itself with console.log ("Re-enqueued N active run(s) on startup"). With an OpenTelemetry setup that instruments a logger rather than the console, there is no signal that recovery ran or what it found.

Evidence

A run interrupted by a deploy at 09:53 made no progress for 31 minutes. It resumed within 30 seconds of an unrelated deploy at 10:25, whose boot sweep minted a fresh job. Nothing in between would have recovered it.

What we would like

A periodic recovery pass, or a documented supported way to run the sweep from application code so we can schedule it ourselves.

reenqueueActiveRuns is importable from @workflow/world today, but calling it against a live world is not a documented contract, and it re-enqueues every pending/running run indiscriminately — which is right at boot and too blunt on a timer, since a run that is legitimately mid-step would be re-enqueued alongside the stranded ones.

Related: #3992 (attempts are consumed by lost workers rather than failures, which is what strands the run in the first place) and #3994 (the retry cap is not configurable).