Repository navigation
[world-postgres] Scope startup recovery to the World's jobPrefix - #4761
Merged
Merged
Conversation
`start()` re-enqueued every active run in the database into this World's
own `${jobPrefix}flows` task, whatever prefix the run was created under,
so the World's runner then drove it. Apps that share a database and keep
their live jobs apart with distinct `jobPrefix` values still took over
each other's active runs on every boot.
Each run is now stamped with the effective `jobPrefix` of the World that
created it (new nullable `workflow_runs.job_prefix`, migration 0026), on
both creation paths: an explicit `run_created` and the resilient
`run_started` that recreates a run from its queued message. `start()`
lists only the active runs stamped with its own prefix, plus unstamped
runs (created before the column existed, or by storage built without a
World), which every World keeps recovering as before. The public
`runs.list` stays unscoped, and the stamp is stripped from returned runs.
The queue's task names and the stamp share one `resolveJobPrefix()`, so
a recovered run always lands in the task whose runner claims it.
Refs #2978 (the recovery half).
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…databases The Postgres World page said `WORKFLOW_POSTGRES_JOB_PREFIX` was useful when sharing a database, but recovery ignored it. Say what it now does: a World claims and recovers only its own prefix's runs, runs from before migration 0026 are recovered by every World, and the topic namespace does not separate apps. Give the SQL that assigns unstamped runs to one app or moves active runs after a prefix rename. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: 4c54f84 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 4008 | 0 | 1392 | 5400 |
| ✅ 💻 Local Development | 5249 | 0 | 551 | 5800 |
| ✅ 📦 Local Production | 5249 | 0 | 551 | 5800 |
| ✅ 🐘 Local Postgres | 5249 | 0 | 551 | 5800 |
| ✅ 🪟 Windows | 388 | 0 | 12 | 400 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 107 | 175 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 903 | 0 | 297 | 1200 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-spec9 | 344 | 0 | 56 | 400 |
| ✅ vercel-ws-transport | 611 | 0 | 189 | 800 |
| Total | 22096 | 0 | 3706 | 25802 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 146 | 0 | 54 |
| ✅ astro-quickjs | 146 | 0 | 54 |
| ✅ example-node | 146 | 0 | 54 |
| ✅ example-quickjs | 146 | 0 | 54 |
| ✅ express-node | 146 | 0 | 54 |
| ✅ express-quickjs | 146 | 0 | 54 |
| ✅ fastify-node | 146 | 0 | 54 |
| ✅ fastify-quickjs | 146 | 0 | 54 |
| ✅ hono-node | 146 | 0 | 54 |
| ✅ hono-quickjs | 146 | 0 | 54 |
| ✅ nest-node | 146 | 0 | 54 |
| ✅ nest-quickjs | 146 | 0 | 54 |
| ✅ nextjs-turbopack-node | 173 | 0 | 27 |
| ✅ nextjs-turbopack-quickjs | 173 | 0 | 27 |
| ✅ nextjs-webpack-node | 173 | 0 | 27 |
| ✅ nextjs-webpack-quickjs | 173 | 0 | 27 |
| ✅ nitro-node | 146 | 0 | 54 |
| ✅ nitro-quickjs | 146 | 0 | 54 |
| ✅ nuxt-node | 146 | 0 | 54 |
| ✅ nuxt-quickjs | 146 | 0 | 54 |
| ✅ python-node | 66 | 0 | 134 |
| ✅ sveltekit-node | 165 | 0 | 35 |
| ✅ sveltekit-quickjs | 165 | 0 | 35 |
| ✅ tanstack-start-node | 146 | 0 | 54 |
| ✅ tanstack-start-quickjs | 146 | 0 | 54 |
| ✅ vite-node | 146 | 0 | 54 |
| ✅ vite-quickjs | 146 | 0 | 54 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 171 | 0 | 29 |
| ✅ astro-stable-quickjs | 171 | 0 | 29 |
| ✅ express-stable-node | 171 | 0 | 29 |
| ✅ express-stable-quickjs | 171 | 0 | 29 |
| ✅ fastify-stable-node | 171 | 0 | 29 |
| ✅ fastify-stable-quickjs | 171 | 0 | 29 |
| ✅ hono-stable-node | 171 | 0 | 29 |
| ✅ hono-stable-quickjs | 171 | 0 | 29 |
| ✅ nest-stable-node | 171 | 0 | 29 |
| ✅ nest-stable-quickjs | 171 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 199 | 0 | 1 |
| ✅ nitro-stable-node | 171 | 0 | 29 |
| ✅ nitro-stable-quickjs | 171 | 0 | 29 |
| ✅ nuxt-stable-node | 171 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 171 | 0 | 29 |
| ✅ sveltekit-stable-node | 190 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 190 | 0 | 10 |
| ✅ tanstack-start-node | 171 | 0 | 29 |
| ✅ tanstack-start-quickjs | 171 | 0 | 29 |
| ✅ vite-stable-node | 171 | 0 | 29 |
| ✅ vite-stable-quickjs | 171 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 171 | 0 | 29 |
| ✅ astro-stable-quickjs | 171 | 0 | 29 |
| ✅ express-stable-node | 171 | 0 | 29 |
| ✅ express-stable-quickjs | 171 | 0 | 29 |
| ✅ fastify-stable-node | 171 | 0 | 29 |
| ✅ fastify-stable-quickjs | 171 | 0 | 29 |
| ✅ hono-stable-node | 171 | 0 | 29 |
| ✅ hono-stable-quickjs | 171 | 0 | 29 |
| ✅ nest-stable-node | 171 | 0 | 29 |
| ✅ nest-stable-quickjs | 171 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 199 | 0 | 1 |
| ✅ nitro-stable-node | 171 | 0 | 29 |
| ✅ nitro-stable-quickjs | 171 | 0 | 29 |
| ✅ nuxt-stable-node | 171 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 171 | 0 | 29 |
| ✅ sveltekit-stable-node | 190 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 190 | 0 | 10 |
| ✅ tanstack-start-node | 171 | 0 | 29 |
| ✅ tanstack-start-quickjs | 171 | 0 | 29 |
| ✅ vite-stable-node | 171 | 0 | 29 |
| ✅ vite-stable-quickjs | 171 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 171 | 0 | 29 |
| ✅ astro-stable-quickjs | 171 | 0 | 29 |
| ✅ express-stable-node | 171 | 0 | 29 |
| ✅ express-stable-quickjs | 171 | 0 | 29 |
| ✅ fastify-stable-node | 171 | 0 | 29 |
| ✅ fastify-stable-quickjs | 171 | 0 | 29 |
| ✅ hono-stable-node | 171 | 0 | 29 |
| ✅ hono-stable-quickjs | 171 | 0 | 29 |
| ✅ nest-stable-node | 171 | 0 | 29 |
| ✅ nest-stable-quickjs | 171 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 199 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 199 | 0 | 1 |
| ✅ nitro-stable-node | 171 | 0 | 29 |
| ✅ nitro-stable-quickjs | 171 | 0 | 29 |
| ✅ nuxt-stable-node | 171 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 171 | 0 | 29 |
| ✅ sveltekit-stable-node | 190 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 190 | 0 | 10 |
| ✅ tanstack-start-node | 171 | 0 | 29 |
| ✅ tanstack-start-quickjs | 171 | 0 | 29 |
| ✅ vite-stable-node | 171 | 0 | 29 |
| ✅ vite-stable-quickjs | 171 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 194 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 194 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 107 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 146 | 0 | 54 |
| ✅ express | 146 | 0 | 54 |
| ✅ hono | 146 | 0 | 54 |
| ✅ nextjs-turbopack | 173 | 0 | 27 |
| ✅ nitro | 146 | 0 | 54 |
| ✅ vite | 146 | 0 | 54 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-spec9
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 172 | 0 | 28 |
| ✅ nextjs-turbopack-quickjs | 172 | 0 | 28 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 146 | 0 | 54 |
| ✅ express | 146 | 0 | 54 |
| ✅ nextjs-turbopack | 173 | 0 | 27 |
| ✅ vite | 146 | 0 | 54 |
Contributor
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 150495ms → this run 93674ms (Δ -56821ms, -38%) 📜 Previous results (6)b6efee6Fri, 30 Oct 2026 18:36:53 GMT · run logs
96c3e47Fri, 30 Oct 2026 16:02:32 GMT · run logs
188e167Thu, 15 Oct 2026 03:00:50 GMT · run logs
890b672Thu, 08 Oct 2026 16:10:15 GMT · run logs
6e5aa33Thu, 08 Oct 2026 06:30:30 GMT · run logs
1a68149Thu, 08 Oct 2026 06:03:36 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 45 total
Full trace: |
Contributor
About these numbersSizes are gzip; parentheses show the change against
|
…t by migration number A run has no job_prefix when an earlier version of the World created it, including old code still running after the migration was applied, so "created before migration 0026" was imprecise. Dropping the number also keeps the docs right if the migration is renumbered when another open world-postgres migration (#4757 also claims 0026) lands first. The changeset is trimmed to two sentences. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…he environment The CLI and the observability UI build their World from the environment, so a run they create (a re-run) is stamped with the prefix they see. With recovery scoped to the prefix, an app that sets jobPrefix only in code no longer has such a run swept into its own task on its next boot. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * origin/main: [deps] Upgrade astro to 7.2.10 for GHSA-26w7-cxv4-gfx2 (#4738) [deps] Upgrade next to 16.3.8 for CVE-2026-94483 (#4737)
2 of 3 tasks
#4757 also adds a migration 0026, with the same journal "when". Drizzle's migrator applies only journal entries whose "when" is later than the last migration recorded in the database, so two entries with one timestamp cannot both land: whichever is applied second is skipped silently. This migration becomes 0027, with a "when" one day after #4757's. #4757 has to merge first; this branch then merges main and keeps both journal entries. In the other order, databases that ran 0027 would skip 0026. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2 of 3 tasks
2 of 3 tasks
pranaygp
marked this pull request as ready for review
October 8, 2026 16:20
VaguelySerious
approved these changes
Oct 9, 2026
pranaygp
added a commit
that referenced
this pull request
Oct 15, 2026
) ### Summary `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. ### Why - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. ### What changed - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. ### Compatibility and rollout - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. ### How did you test your changes? New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. ### Docs Preview | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | ### Open questions for reviewers 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. ### Related PRs and merge order - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. ### PR Checklist - Required to merge - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
main now has 0026_add_seq_in_band and 0027_timestamps_with_time_zone, so the job_prefix migration becomes 0028 with a later journal `when`. Recovery's scoped runs list picks up main's `sortOrder`, and the jobPrefix stamping moves into main's unfenced events storage, which the in-band fence wraps. The migration uses `ADD COLUMN IF NOT EXISTS`, like 0026, since two concurrent `bootstrap` processes can both apply it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
github-actions Bot
added a commit
that referenced
this pull request
Oct 15, 2026
) `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
|
Bump! |
…art-scope Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
VaguelySerious
approved these changes
Oct 30, 2026
VaguelySerious
enabled auto-merge (squash)
October 30, 2026 18:24
VaguelySerious
disabled auto-merge
October 30, 2026 18:51
Renumber the job_prefix migration to 0029 after main's 0028_add_start_hook_admissions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
No backport to This fixes real cross-app run takeover during To override, re-run the Backport to stable workflow manually via |
Merged
1 task done
karthikscale3
pushed a commit
that referenced
this pull request
Nov 2, 2026
…5180) ### Description Follow-up to #5091. That PR documented passing a stored `Hook` to `resumeHook()` on the `resumeHook` page only. This adds short mentions where users and World authors look for it: - **`resumeHook`**: the intro mentions that you can pass a `Hook`, and the reuse section says the Vercel World supports guarded delivery, while other Worlds deliver a Hook object through their existing path. - **`getHookByToken`**: store the returned Hook and pass it to `resumeHook()` for repeated deliveries. - **What's new in v5**: one bullet under *Also new in 5.0*. - **Upgrading a World to v5**: a `Resolved hook resume` row in the optional-surface table for `capabilities.resolvedHookResume`. I also checked #5156 and #4761, which shipped their own docs updates. No stale references to `start({ hook })` being refused on spec 9 remain. ### Docs Preview - [resumeHook: Reuse a resolved hook](https://workflow-docs-git-workflow-docs-cleanup-mon-2-nov.vercel.sh/docs/api-reference/workflow-api/resume-hook#reuse-a-resolved-hook) - [getHookByToken](https://workflow-docs-git-workflow-docs-cleanup-mon-2-nov.vercel.sh/docs/api-reference/workflow-api/get-hook-by-token) - [What's new in v5: Also new in 5.0](https://workflow-docs-git-workflow-docs-cleanup-mon-2-nov.vercel.sh/docs/whats-new#also-new-in-50) - [Upgrading a World to v5: New optional surface](https://workflow-docs-git-workflow-docs-cleanup-mon-2-nov.vercel.sh/worlds/upgrading-to-v5#new-optional-surface) ### PR Checklist - Required to merge - [x] 🔒 DCO sign-off included on every commit 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
world-postgres'sstart()re-enqueues every active run in the database into its own${jobPrefix}flowsGraphile task, and its own runner then drives them.jobPrefixis the documented knob for sharing a database, and it does keep live jobs apart, because a runner claims only its own task. Startup recovery ignores it, so two apps with distinct prefixes still take over each other's active runs on every boot.This PR stamps each run with the prefix of the World that created it and scopes
start()'s recovery tojob_prefix = <own prefix> OR job_prefix IS NULL. Single-app installs see no change.Refs #2978 (the recovery half only).
Why
jobPrefixvalues ("Useful when sharing a database between multiple applications"). With today's recovery, that only separates them until one of them restarts.namespace(WORKFLOW_QUEUE_NAMESPACE) does not help either: world-postgres uses it only for the topic prefix, which it drops at enqueue.WorkflowNotRegisteredErroris classifiedRUNTIME_ERROR, thenrun_failed. This is inferred from the code; I did not reproduce it. When the sets overlap, the wrong app runs the workflow. In Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs #2978, agents ran each other's turns with the wrong credentials.The behaviour is unchanged on
mainsince 5.0.2.What changed
All in
packages/world-postgres:workflow_runs.job_prefix varcharcolumn, migration0028_add_run_job_prefix(ADD COLUMN IF NOT EXISTS, since two concurrentbootstraps can both apply it).storage.ts):createEventsStorage(drizzle, { jobPrefix })stamps both run-creation paths: an explicitrun_created, and the resilientrun_startedthat recreates a run from its queued message.storage.ts,index.ts):runs.listis extracted intocreateRunsList(drizzle, scope?), and a newcreateRecoverableRunsList(drizzle, jobPrefix)adds thejob_prefix = $1 OR job_prefix IS NULLpredicate.start()passesreenqueueActiveRunsa runs storage whoselistis the scoped one.config.ts,queue.ts): a newresolveJobPrefix(config)(config.jobPrefix || 'workflow_') feeds both the Graphile task names and the stamp, so a recovered run always lands in the task whose runner claims it.runs.list(CLI, dashboard) stays unscoped, andjobPrefixis stripped from every run storage returns. There is no change to the@workflow/worldinterface. The shape follows world-local's scoped recovery in [world-local] Scope untagged startup recovery to untagged runs #2667.WORKFLOW_POSTGRES_JOB_PREFIX), Configuration › Worlds (jobPrefix) and the package README now say to give each app sharing a database its own prefix, set it through the environment, that the namespace does not separate apps, and give the SQL that assigns unstamped runs to one app or re-homes runs after a prefix rename.@workflow/world-postgres: minor.Compatibility and rollout
bootstrapbefore deploying this version. Drizzle names every column in its selects and inserts, so new code on an un-migrated database fails withcolumn "job_prefix" does not exist. Old code on a migrated database is fine, because the column is nullable. This is the same contract as the earlier column adds (0016, 0024), and the changeset says so.jobPrefix: runs stamped with the old prefix are no longer swept into the new prefix's task (their live jobs were already orphaned in the old task). The docs give a one-lineUPDATEto re-home them.workflow_by default). Before this PR, the app's next boot swept such a run into its own task; now it stays in a task only a World with that prefix claims. The docs now say to set the prefix throughWORKFLOW_POSTGRES_JOB_PREFIX.How it was tested
packages/world-postgres, unit (pnpm vitest run src): 8 files, 75 tests pass.src/reenqueue.test.tscovers recovery going through the scoped listing rather than the publicruns.list, and the scope and stamp following the default prefix, a configured prefix, andWORKFLOW_POSTGRES_JOB_PREFIXthrough the default config. Against the oldindex.ts, 5 of its 16 cases fail.test/recovery-scope.test.ts. Anapp_a_World, anapp_b_World and prefix-less storage each create one active run. A'sstart()must enqueue exactly {A, legacy} intoapp_a_flowsand B's exactly {B, legacy} intoapp_b_flows, while the publicruns.liststill returns all three. Both creation paths must stamp the effective prefix, and the returned run must not carryjobPrefix. Onmainthis fails: A'sstart()enqueued 3 runs, B's included.pnpm test(20 files, 327 tests, testcontainerspostgres:15-alpine). After the renumber and the merge ofmain, all 12test/files (252 tests) pass against a local Postgres 14, with the testcontainers module pointed at that server. Together with the unit tests that is again 327/327.job_prefix varchar. After mergingmain(with [world] In-band writer fence for world-local and world-postgres #4821's0026and fix(world-postgres): store timestamps as timestamp with time zone #4757's0027):@workflow/world-postgres394/394, including fix(world-postgres): store timestamps as timestamp with time zone #4757's journal-order test and its concurrent-bootstrap test (which failed on a plainADD COLUMNbefore theIF NOT EXISTS); typecheck and changeset check clean.pnpm typecheck,pnpm exec biome ci .(0 errors),node scripts/check-changesets.mjs,node scripts/check-no-unrun-tests.mjs, and docsbun ./scripts/lint.tspass.Open questions for reviewers
NULLas the default prefixworkflow_, which isolates legacy runs sooner but strands the open runs of any app that already uses a custom prefix until someone runs the documentedUPDATE. I chose "nothing is stranded". OK?jobPrefixis internal routing, so it is stripped from returned runs and kept out of@workflow/world. Would you rather expose it on the run, for example in the CLI, to make a misconfigured prefix easier to spot?minor(a behaviour change plus a migration). Happy to make itpatch.Related PRs and merge order
seq_in_band) have merged as0027and0026; this branch is merged withmainand its migration is0028.sortOrderinruns.list) is onmain;createRunsList(and so the recovery list) carries itssortOrder.start()or its tests, so either order works.start()andreenqueue.test.tsand compose with this one: feat(world-postgres): make startup recovery optional #3690 (an on/off switch for recovery), fix(world-postgres): skip parked runs and dedupe recovery jobs on startup #3162 (skip parked runs, dedupe) and [world-postgres] Add a producer role that enqueues without a runner or startup recovery #3969 (producer role). Expect small mechanical conflicts.Docs Preview
WORKFLOW_POSTGRES_JOB_PREFIXjobPrefixPR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete🤖 Generated with Claude Code