Repository navigation
fix(world-postgres): store timestamps as timestamp with time zone - #4757
Conversation
Every timestamp column except hooks.token_retention_until was `timestamp without time zone`. `DEFAULT now()` stored the session's local wall time while drizzle reads non-tz values as UTC, so on a server (or session) outside UTC every createdAt was off by the UTC offset: 7 hours early on America/Los_Angeles, 5.5 hours late on Asia/Kolkata. createdAt feeds the workflow clock, so Date.now() in workflow code was off too. Columns are now timestamptz. Migration 0026 converts them with the session in UTC, which Postgres 12+ does without rewriting tables, and on non-UTC servers moves values that came from now() (created_at, updated_at on rows never updated, invocations.responded_at) from local wall time to the instant they recorded. `workflow.legacy_timezone` overrides the zone for apps whose pool used a different session time zone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…zone Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 89795ed The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 110356ms → this run 104382ms (Δ -5974ms, -5%) 📜 Previous results (3)30441d8Thu, 08 Oct 2026 16:36:14 GMT · run logs
e2c9c56Thu, 08 Oct 2026 06:19:19 GMT · run logs
0b579eeThu, 08 Oct 2026 06:01:40 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3982 | 0 | 878 | 4860 |
| ✅ 💻 Local Development | 4499 | 0 | 541 | 5040 |
| ✅ 📦 Local Production | 4669 | 0 | 551 | 5220 |
| ✅ 🐘 Local Postgres | 4669 | 0 | 551 | 5220 |
| ✅ 🪟 Windows | 348 | 0 | 12 | 360 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 87 | 155 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 897 | 0 | 183 | 1080 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 607 | 0 | 113 | 720 |
| Total | 19766 | 0 | 2916 | 22682 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 145 | 0 | 35 |
| ✅ astro-quickjs | 145 | 0 | 35 |
| ✅ example-node | 145 | 0 | 35 |
| ✅ example-quickjs | 145 | 0 | 35 |
| ✅ express-node | 145 | 0 | 35 |
| ✅ express-quickjs | 145 | 0 | 35 |
| ✅ fastify-node | 145 | 0 | 35 |
| ✅ fastify-quickjs | 145 | 0 | 35 |
| ✅ hono-node | 145 | 0 | 35 |
| ✅ hono-quickjs | 145 | 0 | 35 |
| ✅ nest-node | 145 | 0 | 35 |
| ✅ nest-quickjs | 145 | 0 | 35 |
| ✅ nextjs-turbopack-node | 172 | 0 | 8 |
| ✅ nextjs-turbopack-quickjs | 172 | 0 | 8 |
| ✅ nextjs-webpack-node | 172 | 0 | 8 |
| ✅ nextjs-webpack-quickjs | 172 | 0 | 8 |
| ✅ nitro-node | 145 | 0 | 35 |
| ✅ nitro-quickjs | 145 | 0 | 35 |
| ✅ nuxt-node | 145 | 0 | 35 |
| ✅ nuxt-quickjs | 145 | 0 | 35 |
| ✅ python-node | 66 | 0 | 114 |
| ✅ sveltekit-node | 164 | 0 | 16 |
| ✅ sveltekit-quickjs | 164 | 0 | 16 |
| ✅ tanstack-start-node | 145 | 0 | 35 |
| ✅ tanstack-start-quickjs | 145 | 0 | 35 |
| ✅ vite-node | 145 | 0 | 35 |
| ✅ vite-quickjs | 145 | 0 | 35 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 151 | 0 | 29 |
| ✅ astro-stable-quickjs | 151 | 0 | 29 |
| ✅ express-stable-node | 151 | 0 | 29 |
| ✅ express-stable-quickjs | 151 | 0 | 29 |
| ✅ fastify-stable-node | 151 | 0 | 29 |
| ✅ fastify-stable-quickjs | 151 | 0 | 29 |
| ✅ hono-stable-node | 151 | 0 | 29 |
| ✅ hono-stable-quickjs | 151 | 0 | 29 |
| ✅ nest-stable-node | 151 | 0 | 29 |
| ✅ nest-stable-quickjs | 151 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-turbopack-quickjs-snapshot | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 179 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 179 | 0 | 1 |
| ✅ nitro-stable-node | 151 | 0 | 29 |
| ✅ nitro-stable-quickjs | 151 | 0 | 29 |
| ✅ nuxt-stable-node | 151 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 151 | 0 | 29 |
| ✅ sveltekit-stable-node | 170 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 170 | 0 | 10 |
| ✅ tanstack-start-node | 151 | 0 | 29 |
| ✅ tanstack-start-quickjs | 151 | 0 | 29 |
| ✅ vite-stable-node | 151 | 0 | 29 |
| ✅ vite-stable-quickjs | 151 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 174 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 174 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 87 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 35 |
| ✅ express | 145 | 0 | 35 |
| ✅ hono | 145 | 0 | 35 |
| ✅ nextjs-turbopack | 172 | 0 | 8 |
| ✅ nitro | 145 | 0 | 35 |
| ✅ vite | 145 | 0 | 35 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 145 | 0 | 35 |
| ✅ express | 145 | 0 | 35 |
| ✅ nextjs-turbopack | 172 | 0 | 8 |
| ✅ vite | 145 | 0 | 35 |
Sim WorldSimulated world deterministic testing for races. Traces 🟢 world-sim scenario book — 0 fail of 42 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0026. Unlike earlier migrations, a second application does not fail: the ALTERs are no-ops and the repair UPDATE moves created_at by the server's offset again (7 h into the future on America/Los_Angeles). Lock the tables first and skip the migration when workflow_runs.created_at is already timestamptz. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t by migration number A run has no job_prefix when an earlier version of the World created it, including old code still running after the migration was applied, so "created before migration 0026" was imprecise. Dropping the number also keeps the docs right if the migration is renumbered when another open world-postgres migration (#4757 also claims 0026) lands first. The changeset is trimmed to two sentences. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * origin/main: [deps] Upgrade astro to 7.2.10 for GHSA-26w7-cxv4-gfx2 (#4738) [deps] Upgrade next to 16.3.8 for CVE-2026-94483 (#4737)
…servers The docs say migrations can run as a post-deployment script. On a server outside UTC, migration 0026 also rewrites created_at values, and older versions read the converted columns shifted by the server's offset, so stop the previous version's workers before running it. Also mention the workflow.legacy_timezone override. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#4757 also adds a migration 0026, with the same journal "when". Drizzle's migrator applies only journal entries whose "when" is later than the last migration recorded in the database, so two entries with one timestamp cannot both land: whichever is applied second is skipped silently. This migration becomes 0027, with a "when" one day after #4757's. #4757 has to merge first; this branch then merges main and keeps both journal entries. In the other order, databases that ran 0027 would skip 0026. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…imezone test pool.end() resolves before its connections finish closing, so stopping the test container can still deliver "terminating connection due to administrator command" to a closing client. pg-pool re-emits it on the pool, and with no listener vitest reports an uncaught exception and fails the Unit Tests job even though every test passed. Swallow only that code on the test's pools. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * origin/main: [world-vercel] In-band writer fence params and InBandSupersededError (#4733)
| { | ||
| "idx": 26, | ||
| "version": "7", | ||
| "when": 1789948800000, |
There was a problem hiding this comment.
Agent find: [P1] Give this migration a new number and a later when before merging. Current main already contains 0026_add_seq_in_band from #4821 with the same when: 1789948800000. Drizzle selects pending migrations by this timestamp, so a database upgraded through current main silently skips this conversion. I reproduced this against PostgreSQL 15: applying current main, then this PR's migrations, reports success while workflow_runs.created_at remains timestamp without time zone. A control using 0027 with a later when applies the conversion correctly. Please rebase, use the next available migration number and timestamp, and add an upgrade test starting from current main.
There was a problem hiding this comment.
Fixed in 89795ed: merged main and renumbered to 0027_timestamps_with_time_zone with a when one day after main's 0026_add_seq_in_band. The upgrade tests now start from main's last migration (0026 is the last naive one), and a new journal test checks every entry's idx matches its file number and its when is strictly later than the previous one; restoring the duplicate when fails it along with every upgrade test. Note #4761 also uses 0027 with the same when, so whichever lands second needs 0028; that test catches it.
|
|
||
| <Callout type="warn"> | ||
| Migration 0026 converts every timestamp column to `timestamp with time zone`. | ||
| On a server whose time zone is UTC (the default on most hosted providers) it |
There was a problem hiding this comment.
Agent find: [P2] Expand the migration-before-deployment warning to cover non-UTC app hosts even when the database runs in UTC. The new withTimezone: true reader parses offset-less values from an unmigrated database in the Node process's local time zone. I reproduced this with a UTC PostgreSQL database at migration 0025 and TZ=America/Los_Angeles: a retryAfter written as 2026-10-14T17:28:36.692Z reads back as 2026-10-15T00:28:36.692Z, stretching the retry deadline by seven hours. Migrating before starting the new reader preserves the exact instant. The existing post-deployment migration guidance and this server-only warning leave that upgrade path exposed; document the required ordering for this case too, or provide a reader that safely handles the pre-migration schema.
There was a problem hiding this comment.
Fixed in the reader rather than only the docs (89795ed): timestamptz is now a custom type that parses Postgres' offset-suffixed output as is and reads a value with no offset (only an unconverted column prints one) as UTC, as 5.0.x did. New test: unconverted UTC database, process.env.TZ = 'America/Los_Angeles', retryAfter/startedAt/createdAt read back as written; with drizzle's stock reader it fails by 7 h. So on a UTC server the upgrade order no longer matters, and the docs now say so; the ordering stays required only on non-UTC servers.
karthikscale3
left a comment
There was a problem hiding this comment.
Approving: a careful fix for a real bug (event.createdAt drives the workflow clock, so this was more than cosmetic). Nothing blocking; the inline comments are a few risks and nits.
Agent find: verification done on this head (30441d8):
test/timezone.test.tspasses (9/9), and the whole@workflow/world-postgressuite passes (330/330).- With
main's schema and journal put back, the three "keeps createdAt correct" tests fail, so the new test catches the regression. - Hand repro on Postgres 16 in
America/Los_Angeles: applied 0000–0025, wrote rows the way 5.0.x did, applied 0026. Rows fromnow()today, winter rows (repaired by 8h, the standard-time offset), and runs with a JavaScript-writtenupdated_atall came out right, and the session time zone was restored. - Checked: every
INSERTexcept snapshots leavescreated_attoDEFAULT now(); the only SQLnow()writes are invocationresponded_at/expired_at; none of the 8 tables has a trigger; drizzle runs migrations in one transaction, so the lock plus type check does make a concurrent second apply a no-op. - Older versions on a UTC session still read converted columns correctly, so a rolling upgrade is safe there; on non-UTC sessions they read shifted values, as the docs warn.
The ones I'd most like to see before merge (optional): a lock_timeout in 0026, and documenting ALTER DATABASE/ROLE SET workflow.legacy_timezone as an alternative to options.
| current_setting('TimeZone') | ||
| ); | ||
| BEGIN | ||
| LOCK TABLE |
There was a problem hiding this comment.
Agent find: No lock_timeout before taking ACCESS EXCLUSIVE on 8 hot tables. Even in the UTC, catalog-only case, LOCK TABLE waits for every open transaction on these tables, and every new query on them queues behind it while it waits, so a single long transaction can stall the whole World. Consider SET LOCAL lock_timeout = '…' (e.g. via set_config('lock_timeout', '10s', true)) so bootstrap fails fast and can be retried, or mention the risk in the docs.
There was a problem hiding this comment.
Added in 89795ed: lock_timeout is set to 10s for the LOCK TABLE (only when the session's is 0, so an explicit one is respected) and restored right after. Covered by a test that holds a lock on workflow_events from another transaction: bootstrap fails with 55P03, then succeeds once the transaction ends. README and docs mention it.
| ) THEN | ||
| -- An `updated_at` equal to `created_at` came from the same `now()` as it | ||
| -- (the row was never updated). Every other `updated_at` is from JavaScript. | ||
| UPDATE "workflow"."workflow_runs" SET |
There was a problem hiding this comment.
Agent find: On a non-UTC server, the repair is one unbatched UPDATE of every row in each table (including workflow_events, usually the largest) under the exclusive lock. That writes a new version of every row (table bloat, a burst of WAL) and can hold the lock for a long time on big databases. The docs say the locks are held "until it finishes"; it may help to say roughly what it costs (about one rewrite of each table) so operators can plan downtime.
There was a problem hiding this comment.
Documented in 89795ed (migration comment, README, docs callout): about one rewrite of each table, with new row versions, WAL and bloat until vacuum, under the locks, so plan a maintenance window on large databases. Batching outside the lock is listed as an open question in the description.
| ALTER TABLE "workflow"."workflow_stream_chunks" | ||
| ALTER COLUMN "created_at" TYPE timestamp with time zone; | ||
|
|
||
| IF lower(legacy_zone) NOT IN ( |
There was a problem hiding this comment.
Agent find: Nit: this UTC-alias list is only a shortcut. Correctness doesn't depend on it, because the WHERE timezone(...) <> ... filters below skip rows that wouldn't move (I checked <+00>-00 and other zero-offset zones). A one-line comment saying so would stop someone from "fixing" or extending it later.
There was a problem hiding this comment.
Added a comment in 89795ed saying the list is only a shortcut and the WHERE clauses make any zero-offset zone a no-op.
| -- value is taken as UTC, which is already right for every column written | ||
| -- from JavaScript. | ||
| -- 2. When the zone the defaults were written in is not UTC, move each value | ||
| -- that came from `now()` from that zone's wall time to the instant it |
There was a problem hiding this comment.
Agent find: Rows written during the autumn clock-change hour (e.g. 01:00–02:00 on 2026-11-01 in America/Los_Angeles, which happens twice) are resolved as standard time, so they can be off by an hour. I reproduced it: 2026-11-01 01:30 became 09:30Z. That's unavoidable with naive input; worth a line in this comment. (Other daylight-saving handling is right: a winter row 2026-01-15 10:00 correctly moved by 8h, not 7h.)
There was a problem hiding this comment.
Added to the comment in 89795ed: values from the repeated fall-back hour are ambiguous and taken as standard time, so a first-pass row lands an hour late.
| finishes. It assumes those values were written in the time zone of the | ||
| session that runs the migration. If your application's pool set a different | ||
| session time zone, pass it to the migration, for example with | ||
| `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin` in the connection |
There was a problem hiding this comment.
Agent find: The override is documented only as a connection-string options parameter. Some transaction-mode poolers (PgBouncer and several hosted poolers) reject or drop startup options. Suggest also documenting ALTER DATABASE … SET workflow.legacy_timezone = '…' or ALTER ROLE … SET …, which both work for a custom setting like this and are picked up by current_setting('workflow.legacy_timezone', true). The same applies to the docs callout.
There was a problem hiding this comment.
Documented ALTER DATABASE … SET workflow.legacy_timezone / ALTER ROLE … SET in the README, the docs callout and the migration comment (89795ed), with a test that sets it via ALTER DATABASE.
|
|
||
| Migration 0026 converts the columns. On a server in UTC it only changes the | ||
| catalog, without rewriting tables. Otherwise it also rewrites every | ||
| `created_at` row (and `updated_at` on rows never updated) from the server's |
There was a problem hiding this comment.
Agent find: Nit: workflow_invocations.responded_at is rewritten too (migration line 127), but this paragraph lists only created_at and never-updated updated_at.
There was a problem hiding this comment.
Fixed in 89795ed: the README lists workflow_invocations.responded_at too.
| <Callout type="warn"> | ||
| Migration 0026 converts every timestamp column to `timestamp with time zone`. | ||
| On a server whose time zone is UTC (the default on most hosted providers) it | ||
| only changes the catalog. On a server in another time zone it also rewrites |
There was a problem hiding this comment.
Agent find: Not quite exact: on a UTC server the table storage files are unchanged, but the partial index workflow_invocations_pending (its WHERE uses responded_at) is rebuilt. I confirmed its storage file changes across 0026. It's tiny, so this doesn't matter in practice, but "only changes the catalog" (here and in the README) slightly overstates it.
There was a problem hiding this comment.
Fixed in 89795ed: README and docs now say it changes the catalog and rebuilds the small workflow_invocations_pending index, with no table rewrite.
| * until the first update) from `DEFAULT now()`, and every other | ||
| * timestamp from a JavaScript `Date` sent as an ISO string. | ||
| */ | ||
| async function writeLegacyRows(pool: Pool) { |
There was a problem hiding this comment.
Agent find: Test gap (non-blocking): every legacy row is written "now", so nothing covers rows written under a different daylight-saving offset than the one the migration runs in. It works (I checked by hand), but adding a row with a fixed winter created_at, e.g. '2026-01-15 10:00:00' on the Los Angeles database, expected at 18:00Z, would pin down that timezone(legacy_zone, …) applies each row's own offset.
There was a problem hiding this comment.
Added in 89795ed: the legacy rows include an event with created_at 2026-01-15 10:00 in the writing session's zone, expected at its instant (18:00Z on the Los Angeles database), in every upgrade test.
| expectSameInstant(snapshot.createdAt, js); | ||
| } | ||
|
|
||
| async function heapFiles(pool: Pool) { |
There was a problem hiding this comment.
Agent find: Nit: this only compares table storage files. If you want the test to back up the "only changes the catalog" claim, include the indexes too. workflow_invocations_pending will show up as rebuilt, which is expected.
There was a problem hiding this comment.
Done in 89795ed: the UTC test compares the storage file of every table and index, and expects exactly workflow_invocations_pending to change.
| @@ -0,0 +1,5 @@ | |||
| --- | |||
| '@workflow/world-postgres': patch | |||
There was a problem hiding this comment.
Agent find: Nit: given a data-rewriting migration, an upgrade order that matters on non-UTC servers, and no clean rollback, minor may signal this better than patch. Your call.
There was a problem hiding this comment.
Keeping patch: AGENTS.md asks for patch on bug fixes. The upgrade order, cost and lack of rollback are now in the README and docs.
main shipped its own migration 0026 (0026_add_seq_in_band) with the same journal `when` as this branch's 0026. Renumber the timestamp migration to 0027 with a later `when`, so databases already at main's 0026 still apply it, and treat 0026 as the last migration with naive timestamps in the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
- Read a timestamp printed without an offset as UTC. Only a column 0027 has not converted yet prints one, and drizzle's `withTimezone` reader parsed it in the Node process's local zone, moving a `retryAfter` by hours when the upgraded app started before `bootstrap` in a non-UTC Node process. - Wait at most 10 seconds for the migration's table locks unless the session set its own `lock_timeout`, so a long transaction fails `bootstrap` instead of queueing every query on those tables behind it. - Comment the UTC-alias shortcut, the ambiguous fall-back hour, and the cost of the repair. - Document `ALTER DATABASE/ROLE SET workflow.legacy_timezone` for poolers that drop startup `options`, the repair's cost, the lock timeout, the rebuilt partial index, `responded_at`, and that 0027 cannot be rolled back on a non-UTC server. - Test that journal entries are ordered, a winter row, the rebuilt index, the lock timeout, the pre-migration reader, and the `ALTER DATABASE` setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
) `@workflow/world-postgres` stores every timestamp column except `hooks.token_retention_until` as `timestamp without time zone`. On a Postgres server (or session) that is not in UTC, every `createdAt` the World returns is off by the UTC offset. This PR moves all timestamp columns to `timestamp with time zone`, with migration `0027` converting existing databases and repairing the values that were stored wrong. - `created_at` comes from `DEFAULT now()`, which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (`new Date(value + "+0000")`). - JS-written columns (`started_at`, `completed_at`, `retry_after`, ...) go in as ISO strings whose `Z` Postgres ignores, so they hold UTC wall time and read back correctly. - Result, with the server in `America/Los_Angeles`: every run's and step's `createdAt` is 7 h **before** its `startedAt`. `Asia/Kolkata`: 5.5 h in the future. A pool with `-c TimeZone=Asia/Tokyo` on a UTC server: 9 h in the future. - It is more than cosmetic: `event.createdAt` drives the deterministic workflow clock (`advanceClock(+event.createdAt)` in both runtimes), so `Date.now()` inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones. Found while building a durable agent harness on top of Workflow whose CI checks what `workflow inspect` shows for runs on each World: on world-postgres 5.0.2 with the database in `America/Los_Angeles`, `createdAt` was 7 h before `startedAt` for every run and step, and correct with the server in UTC. Present on `main`; no existing issue or PR covers it. - **`src/drizzle/schema.ts`**: every timestamp column is `timestamp with time zone`, via a small `timestamptz` custom type with a comment on why. No storage code changes: ISO `Z` strings and `now()` both land as the right instant. TS types stay `Date`. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's own `withTimezone` reader would parse it in the Node process's local zone. - **Migration `0027_timestamps_with_time_zone.sql`** (one `DO` block; restores the session TimeZone it changes): 1. Sets the session to UTC and converts the 21 columns with no `USING` clause. Postgres 12+ skips the table rewrite for `timestamp` -> `timestamptz` when TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column. 2. If the zone the defaults were written in is not a UTC alias, moves values that only ever came from `now()` from that zone's wall time to the instant they recorded: `created_at` on runs, events, steps, hooks, waits, invocations and stream chunks; `updated_at` on runs/steps/waits only where `updated_at = created_at` (never updated, so both came from the same `now()`); `invocations.responded_at`. Left alone: `invocations.expired_at` (mixes `now()` and JS dates, only null-checked) and `snapshots.created_at` (always written from JS). 3. The legacy zone is the migration session's TimeZone, overridable with `workflow.legacy_timezone` (e.g. `?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin`) for apps whose pool set a different session zone than `bootstrap` uses. `ALTER DATABASE/ROLE ... SET workflow.legacy_timezone` works too, for poolers that drop startup `options`. 4. Runs once. The drizzle migrator takes no lock, so two `bootstrap` processes started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes when `workflow_runs.created_at` is already `timestamptz`. It waits at most 10 s for those locks (unless the session set its own `lock_timeout`), so a long transaction fails `bootstrap` instead of queueing every query behind it. - **`meta/_journal.json`**: entry `idx 27`, `when` one day after main's `0026_add_seq_in_band`. - **Package README**: a "Timestamps and time zones" section on the migration and the override. - **Docs** (`docs/content/worlds/v5/postgres.mdx`): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below. - **Changeset**: `@workflow/world-postgres` patch. - **Test hygiene**: the new test's pools ignore `57P01` ("terminating connection due to administrator command"). `pool.end()` resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown. - **UTC servers** (the default on Neon, Supabase, RDS and the Docker image): catalog-only ALTER, milliseconds under ACCESS EXCLUSIVE on the 8 tables, plus a rebuild of the small `workflow_invocations_pending` partial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower). - **Non-UTC servers**: the repair UPDATE rewrites every `created_at` row (events and stream chunks can be large) while holding those locks, once. - **Upgrade window.** New code against an unmigrated DB reads naive timestamps as UTC, as old code did, in any Node time zone. Old code against a migrated DB reads correctly in a UTC session and shifted by the session's offset in a non-UTC one. A shifted read moves more than display: `step_started` compares `retryAfter` with `Date.now()`, so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, run `bootstrap`, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server. - **Assumed zone.** The repair assumes old rows were written in the migration session's zone; it is wrong if the server zone changed over time or the app's pool used another zone (hence `workflow.legacy_timezone`). Wall times in a DST fall-back hour are ambiguous and can land 1 h off. - Consumers of `@workflow/world-postgres/schema` see the column SQL type change; the TS types stay `Date`. New `test/timezone.test.ts` (testcontainers `postgres:15-alpine`, skipped on Windows like the other container tests; per-database time zones are set with `ALTER DATABASE ... SET timezone`): - **Fresh schema** with the database in `America/Los_Angeles` and `Asia/Kolkata`, and a UTC server with a pool setting `TimeZone=Asia/Tokyo`: drive run/step/hook/wait creation through the storage API and assert every `createdAt`/`startedAt` (write results, `runs.get`, `steps.get`, `hooks.get`, `events.list`) is the current instant. - **Schema check**: every timestamp column in `workflow` is `timestamp with time zone` and the `created_at` defaults stay `now()`. - **Upgrade path**: migrate to 0026 (the last naive migration, current `main`) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. In `America/Los_Angeles`, `created_at` and never-updated `updated_at` equal the transaction's real `now()` (and a `created_at` from January is moved by the standard-time offset, not today's), while JS-written `started_at`/`updated_at`/`resume_at`/snapshot `created_at` are unchanged. On a UTC server the same holds and `pg_relation_filenode` of all 8 tables and their indexes is unchanged except `workflow_invocations_pending`, whose predicate reads `responded_at` (no table rewrite). With rows written in a `TimeZone=Asia/Tokyo` session, `workflow.legacy_timezone=Asia/Tokyo` repairs them (without it they stay 9 h off), set either in the connection's `options` or with `ALTER DATABASE`. - **Runs once**: two `runMigrations` in parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms. - **Lock timeout**: with another transaction holding a lock on `workflow_events`, `bootstrap` fails with `55P03` within the 10 s timeout, and succeeds once that transaction ends. - **Reader before the migration**: on an unconverted UTC database with `process.env.TZ = 'America/Los_Angeles'`, `retryAfter`/`startedAt`/`createdAt` read back as the instants written (with drizzle's stock reader `retryAfter` comes back 7 h late). - **Journal order**: every entry's `idx` matches its file number and its `when` is later than the one before, so a duplicate `when` like this branch's pre-merge 0026 fails CI. Results: - Latest head (after merging `main` at ecaa047): `test/timezone.test.ts` 13/13 and the whole package 388/388 on `postgres:15-alpine`; typecheck and Biome clean on changed files; `node scripts/check-changesets.mjs` OK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicate `when` fails the journal test and every upgrade test. - Earlier heads: on `main`'s schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 on `postgres:15-alpine`, `postgres:12-alpine` and `postgres:18-alpine` (the image the `e2e-local-postgres` job uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latest `main`. - After merging `main`: `storage.test.ts`, `retention.test.ts` and `run-creation.test.ts` 185/185 on PostgreSQL 14.20; `pnpm vitest run src` 71/71; `pnpm build` and `pnpm typecheck` clean; Biome clean on changed files; `node scripts/check-changesets.mjs` OK. - CI: the `Unit Tests (ubuntu-latest)` job failed twice on this branch with 330/330 tests passing and two unhandled `57P01` errors attributed to `test/timezone.test.ts` (the teardown race above, now handled in this file and reproduced in isolation with `pg_terminate_backend` on an idle pooled client). `main` currently fails the same job the same way from `test/invoke.test.ts` (e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, in `packages/core` (`fanout-dispatch.test.ts` "publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against `< 260`), which this PR does not touch; it needs a rerun to show the world-postgres result. - Earlier full-package runs (`pnpm build && pnpm test`, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff: `storage.test.ts` "should release a retained hook after its deadline" (writes a host `Date.now() - 1` and expects the container's `now()` to be past it, on the already-`timestamptz` `token_retention_until`) and `invoke.test.ts` "slow fallback" (3 s budget under parallel load). - A scratch test (not committed) wrote rows with `main`'s storage code on servers in `America/Los_Angeles`, `Asia/Kolkata`, `Europe/London`, `America/St_Johns` (-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log's `createdAt` stays monotonic across the upgrade, and `main`'s code reading the migrated database from a UTC session gets the same instants. | Page | Preview | | --- | --- | | Postgres World, Installation (migration 0027 callout) | [/worlds/postgres#installation](https://workflow-docs-git-pgp-world-postgres-utc-timestamps.vercel.sh/worlds/postgres#installation) | 1. **`timestamptz` or the keep-`timestamp` fallback?** The fallback: keep `timestamp`, change the defaults to `timezone('utc', now())`, fix the raw `now()` writes in `invocations.ts`, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any future `now()` write. This PR prefers the type change. 2. **Repair under lock.** On non-UTC servers the repair UPDATE runs inside the migration while holding ACCESS EXCLUSIVE on the 8 tables, about one rewrite of each table (now documented so operators can plan a window). Acceptable for a one-time migration, or should the repair be batched outside the lock? 3. **Changeset level.** Kept as `patch`, per AGENTS.md (`patch` for bug fixes); the review suggested `minor` given the data-rewriting migration. - `main` shipped its own `0026_add_seq_in_band` (#4821) with the journal `when` this branch's 0026 had, so this migration is now **`0027`**. Without the renumber, databases already at main's 0026 would skip the conversion silently. - **#4761** (`[world-postgres] Scope startup recovery to the World's jobPrefix`) also adds `0027_add_run_job_prefix` with the same `when` as this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take `0028` and a later `when`; the journal-order test in this PR fails on the clash. - #4756 and #4759 touch world-postgres without migrations and are independent of this one. - [x] 📦 `pnpm changeset` was run to create a changelog for this PR - [x] 🔒 DCO sign-off passes (run `git commit --signoff` on your commits) - [ ] 📝 Ping `@vercel/workflow` in a comment once the PR is ready, and the above checklist is complete 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
|
Backport PR opened against |
…th time zone (#4892) Automated backport of #4757 to `stable` ([backport job run](https://github.com/vercel/workflow/actions/runs/38672264664)). Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @workflow/astro@5.0.6 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.6 - @workflow/rollup@5.0.6 - @workflow/vite@5.0.6 ## @workflow/builders@5.0.6 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/errors@5.0.5 ## @workflow/cli@5.0.6 ### Patch Changes - [#4758](#4758) [`0bc17d7`](0bc17d7) Thanks [@pranaygp](https://github.com/pranaygp)! - Resolve a run's encryption key from metadata only when decrypting in `workflow inspect --decrypt` and the web UI, instead of making the server resolve the run's whole input and output for each lookup. - Updated dependencies [[`91297ab`](91297ab), [`5170f8f`](5170f8f), [`0bc17d7`](0bc17d7), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917), [`a067ef7`](a067ef7)]: - @workflow/core@5.2.3 - @workflow/world-local@5.1.2 - @workflow/web@5.0.6 - @workflow/world@5.0.5 - @workflow/world-vercel@5.2.3 - @workflow/errors@5.0.5 - @workflow/builders@5.0.6 ## @workflow/core@5.2.3 ### Patch Changes - [#4335](#4335) [`91297ab`](91297ab) Thanks [@pranaygp](https://github.com/pranaygp)! - Security: remove the `nanoid` dependency (pinned at 5.1.6, affected by [GHSA-28wg-ghj8-5hjv](GHSA-28wg-ghj8-5hjv)). Default hook tokens now come from a built-in copy of the same generator, so tokens and seeded random values stay identical for runs recorded on earlier versions. Upgrading `nanoid` itself would have changed them, because `nanoid@5.1.16` draws a different number of random bytes per id. - [#4758](#4758) [`0bc17d7`](0bc17d7) Thanks [@pranaygp](https://github.com/pranaygp)! - Resolve a run's encryption key from metadata only, so `Run#getReadable()` and resuming a hook stored without a resume context no longer make the server resolve the run's whole input and output first. - [#4916](#4916) [`42077f2`](42077f2) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Fix cross-deployment `start()` to 4.x deployments being misread as a run invocation, and keep replayed runs in the original run's region. - [#4870](#4870) [`59b76f8`](59b76f8) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Stamp the default snapshot threshold into QuickJS runs at `start()` so they qualify for the event limit exemption, warn when a run approaches the event limit, and recommend `WORKFLOW_VM=quickjs` in `MAX_EVENTS_EXCEEDED` and `REPLAY_TIMEOUT` errors - [#4866](#4866) [`73a7efa`](73a7efa) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Read event logs where a step's creation is recorded on its `step_started` alone, with no separate `step_created`. - [#4874](#4874) [`2163290`](2163290) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Preserve `workflowVm` and `snapshotThreshold` from `executionContext` in `hydrateResourceIO` for observability UIs - [#4895](#4895) [`0d434fc`](0d434fc) Thanks [@karthikscale3](https://github.com/karthikscale3)! - Avoid redundant history reads when independent hooks register concurrently. Keep existing single-event requests parallel and reconcile their committed events together before continuing, with the normal history reload when the responses do not cover a complete consecutive suffix. - [#4819](#4819) [`cd2f917`](cd2f917) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Ignore events a step or hook writes after its outcome during replay instead of failing the run - Updated dependencies [[`5170f8f`](5170f8f), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`0d434fc`](0d434fc), [`a067ef7`](a067ef7)]: - @workflow/world-local@5.1.2 - @workflow/world@5.0.5 - @workflow/world-vercel@5.2.3 - @workflow/errors@5.0.5 ## @workflow/errors@5.0.5 ### Patch Changes - [#4870](#4870) [`59b76f8`](59b76f8) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Add a QuickJS snapshotting hint and docs link to `MaxEventsExceededError` - Updated dependencies [[`42077f2`](42077f2), [`59b76f8`](59b76f8), [`0d434fc`](0d434fc)]: - @workflow/world@5.0.5 ## @workflow/nest@5.1.4 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.6 ## @workflow/next@5.0.6 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/builders@5.0.6 ## @workflow/nitro@5.0.6 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/web@5.0.6 - @workflow/builders@5.0.6 - @workflow/rollup@5.0.6 - @workflow/vite@5.0.6 ## @workflow/nuxt@5.0.6 ### Patch Changes - Updated dependencies []: - @workflow/nitro@5.0.6 ## @workflow/rollup@5.0.6 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.6 ## @workflow/sveltekit@5.0.6 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.6 - @workflow/rollup@5.0.6 - @workflow/vite@5.0.6 ## @workflow/vite@5.0.6 ### Patch Changes - Updated dependencies []: - @workflow/builders@5.0.6 ## @workflow/vitest@5.0.6 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`5170f8f`](5170f8f), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917), [`a067ef7`](a067ef7)]: - @workflow/core@5.2.3 - @workflow/world-local@5.1.2 - @workflow/world@5.0.5 - @workflow/builders@5.0.6 - @workflow/rollup@5.0.6 ## @workflow/web@5.0.6 ### Patch Changes - [#4758](#4758) [`0bc17d7`](0bc17d7) Thanks [@pranaygp](https://github.com/pranaygp)! - Resolve a run's encryption key from metadata only when decrypting in `workflow inspect --decrypt` and the web UI, instead of making the server resolve the run's whole input and output for each lookup. - [#4874](#4874) [`2163290`](2163290) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Recommend the QuickJS VM with snapshotting on runs that failed with `MAX_EVENTS_EXCEEDED` or `REPLAY_TIMEOUT` - Updated dependencies [[`5170f8f`](5170f8f), [`59b76f8`](59b76f8), [`a067ef7`](a067ef7)]: - @workflow/world-local@5.1.2 ## @workflow/web-shared@5.0.6 ### Patch Changes - [#4866](#4866) [`73a7efa`](73a7efa) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Read event logs where a step's creation is recorded on its `step_started` alone, with no separate `step_created`. - [#4874](#4874) [`2163290`](2163290) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Add `getRunLimitGuidance()` for runs that failed with `MAX_EVENTS_EXCEEDED` or `REPLAY_TIMEOUT`, and show the run's workflow VM in the sidebar - [#4819](#4819) [`cd2f917`](cd2f917) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Ignore events a step or hook writes after its outcome during replay instead of failing the run - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/world@5.0.5 ## workflow@5.2.3 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/cli@5.0.6 - @workflow/errors@5.0.5 - @workflow/next@5.0.6 - @workflow/nitro@5.0.6 - @workflow/typescript-plugin@5.0.0 - @workflow/astro@5.0.6 - @workflow/nest@5.1.4 - @workflow/rollup@5.0.6 - @workflow/sveltekit@5.0.6 - @workflow/nuxt@5.0.6 ## @workflow/world@5.0.5 ### Patch Changes - [#4916](#4916) [`42077f2`](42077f2) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Fix cross-deployment `start()` to 4.x deployments being misread as a run invocation, and keep replayed runs in the original run's region. - [#4870](#4870) [`59b76f8`](59b76f8) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Exempt QuickJS runs with a positive snapshot threshold from the default per-run event limit, matching the Vercel World - [#4895](#4895) [`0d434fc`](0d434fc) Thanks [@karthikscale3](https://github.com/karthikscale3)! - Avoid redundant history reads when independent hooks register concurrently. Keep existing single-event requests parallel and reconcile their committed events together before continuing, with the normal history reload when the responses do not cover a complete consecutive suffix. ## @workflow/world-local@5.1.2 ### Patch Changes - [#4922](#4922) [`5170f8f`](5170f8f) Thanks [@NishitaSikka](https://github.com/NishitaSikka)! - Restore missing `run_created` events for interrupted local run creation. - [#4870](#4870) [`59b76f8`](59b76f8) Thanks [@TooTallNate](https://github.com/TooTallNate)! - Exempt QuickJS runs with a positive snapshot threshold from the default per-run event limit, matching the Vercel World - [#4924](#4924) [`a067ef7`](a067ef7) Thanks [@AndrewBarba](https://github.com/AndrewBarba)! - Fix an edge case in `createHook({ experimental_force: true })` where a racing `resumeHook` could be dropped during the handover - Updated dependencies [[`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`0d434fc`](0d434fc)]: - @workflow/world@5.0.5 - @workflow/errors@5.0.5 ## @workflow/world-postgres@5.0.5 ### Patch Changes - [#4883](#4883) [`93396c5`](93396c5) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Write `run_started`, `run_completed`, `run_failed`, `run_cancelled`, `attr_set`, `step_retrying`, `wait_created` and `wait_completed` in one transaction with the entity change they record. A process that died between the two used to leave an entity in a state its event log did not explain; a completed wait with no `wait_completed` made the run retry the wait forever. A wait left in that state by an earlier version now gets its missing event on the next attempt. - [#4882](#4882) [`ecaa047`](ecaa047) Thanks [@dretanacr](https://github.com/dretanacr)! - Write a step's `step_completed` or `step_failed` row update and its event in one transaction, as `step_started` already does. A process that died between the two used to leave a terminal step with no terminal event, so the replayed `step_started` was refused as terminal and the run never advanced. - [#4812](#4812) [`32f213e`](32f213e) Thanks [@pranaygp](https://github.com/pranaygp)! - Keep a checked-out client in the World's own `pg` pool from ending the process when the database drops its connection. - [#4757](#4757) [`7882c81`](7882c81) Thanks [@pranaygp](https://github.com/pranaygp)! - Store timestamps as `timestamp with time zone` so `createdAt` is correct when the Postgres server or session is not in UTC. Migration 0027 converts the columns and repairs `created_at` values written by earlier versions. - Updated dependencies [[`5170f8f`](5170f8f), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`0d434fc`](0d434fc), [`a067ef7`](a067ef7)]: - @workflow/world-local@5.1.2 - @workflow/world@5.0.5 - @workflow/errors@5.0.5 ## @workflow/world-testing@5.0.6 ### Patch Changes - Updated dependencies [[`91297ab`](91297ab), [`0bc17d7`](0bc17d7), [`0bc17d7`](0bc17d7), [`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`73a7efa`](73a7efa), [`2163290`](2163290), [`0d434fc`](0d434fc), [`cd2f917`](cd2f917)]: - @workflow/core@5.2.3 - @workflow/cli@5.0.6 - @workflow/world@5.0.5 - workflow@5.2.3 ## @workflow/world-vercel@5.2.3 ### Patch Changes - [#4916](#4916) [`42077f2`](42077f2) Thanks [@VaguelySerious](https://github.com/VaguelySerious)! - Fix cross-deployment `start()` to 4.x deployments being misread as a run invocation, and keep replayed runs in the original run's region. - [#4895](#4895) [`0d434fc`](0d434fc) Thanks [@karthikscale3](https://github.com/karthikscale3)! - Avoid redundant history reads when independent hooks register concurrently. Keep existing single-event requests parallel and reconcile their committed events together before continuing, with the normal history reload when the responses do not cover a complete consecutive suffix. - Updated dependencies [[`42077f2`](42077f2), [`59b76f8`](59b76f8), [`59b76f8`](59b76f8), [`0d434fc`](0d434fc)]: - @workflow/world@5.0.5 - @workflow/errors@5.0.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
@workflow/world-postgresstores every timestamp column excepthooks.token_retention_untilastimestamp without time zone. On a Postgres server (or session) that is not in UTC, everycreatedAtthe World returns is off by the UTC offset. This PR moves all timestamp columns totimestamp with time zone, with migration0027converting existing databases and repairing the values that were stored wrong.Why
created_atcomes fromDEFAULT now(), which Postgres casts to the session's local wall time for a naive column. drizzle reads naive columns back as UTC (new Date(value + "+0000")).started_at,completed_at,retry_after, ...) go in as ISO strings whoseZPostgres ignores, so they hold UTC wall time and read back correctly.America/Los_Angeles: every run's and step'screatedAtis 7 h before itsstartedAt.Asia/Kolkata: 5.5 h in the future. A pool with-c TimeZone=Asia/Tokyoon a UTC server: 9 h in the future.event.createdAtdrives the deterministic workflow clock (advanceClock(+event.createdAt)in both runtimes), soDate.now()inside workflow code stays pinned at run start for hours on UTC-negative servers and jumps ahead on UTC-positive ones.Found while building a durable agent harness on top of Workflow whose CI checks what
workflow inspectshows for runs on each World: on world-postgres 5.0.2 with the database inAmerica/Los_Angeles,createdAtwas 7 h beforestartedAtfor every run and step, and correct with the server in UTC. Present onmain; no existing issue or PR covers it.What changed
src/drizzle/schema.ts: every timestamp column istimestamp with time zone, via a smalltimestamptzcustom type with a comment on why. No storage code changes: ISOZstrings andnow()both land as the right instant. TS types stayDate. The reader parses Postgres' offset-suffixed output as is, and reads a value with no offset (only an unconverted column prints one) as UTC, as earlier versions did; drizzle's ownwithTimezonereader would parse it in the Node process's local zone.0027_timestamps_with_time_zone.sql(oneDOblock; restores the session TimeZone it changes):USINGclause. Postgres 12+ skips the table rewrite fortimestamp->timestamptzwhen TimeZone is UTC, and stored values are taken as UTC, which is already right for every JS-written column.now()from that zone's wall time to the instant they recorded:created_aton runs, events, steps, hooks, waits, invocations and stream chunks;updated_aton runs/steps/waits only whereupdated_at = created_at(never updated, so both came from the samenow());invocations.responded_at. Left alone:invocations.expired_at(mixesnow()and JS dates, only null-checked) andsnapshots.created_at(always written from JS).workflow.legacy_timezone(e.g.?options=-c%20workflow.legacy_timezone%3DEurope%2FBerlin) for apps whose pool set a different session zone thanbootstrapuses.ALTER DATABASE/ROLE ... SET workflow.legacy_timezoneworks too, for poolers that drop startupoptions.bootstrapprocesses started together both apply 0027, and unlike earlier migrations a second application would not fail: it would move the repaired values by the offset again. The block locks the 8 tables first and returns without changes whenworkflow_runs.created_atis alreadytimestamptz. It waits at most 10 s for those locks (unless the session set its ownlock_timeout), so a long transaction failsbootstrapinstead of queueing every query behind it.meta/_journal.json: entryidx 27,whenone day after main's0026_add_seq_in_band.docs/content/worlds/v5/postgres.mdx): a warning callout under Installation. The page says migrations can run as a post-deployment script; on non-UTC servers 0027 needs the upgrade order below.@workflow/world-postgrespatch.57P01("terminating connection due to administrator command").pool.end()resolves before its sockets close, so stopping the container can still hand that error to a closing client; pg-pool re-emits it on the pool, and with no listener vitest counts an uncaught exception and fails the job even though every test passed. Other errors are still thrown.Compatibility and rollout
workflow_invocations_pendingpartial index. No data changes. Postgres 11 and older rewrite the tables on the ALTER regardless (still correct, slower).created_atrow (events and stream chunks can be large) while holding those locks, once.step_startedcomparesretryAfterwithDate.now(), so a retry backoff can be skipped or stretched by hours. On a non-UTC server: stop the previous version's workers, runbootstrap, then start the upgraded app (the README and the new docs callout say this). For the same reason there is no rollback past 0027 on a non-UTC server.workflow.legacy_timezone). Wall times in a DST fall-back hour are ambiguous and can land 1 h off.@workflow/world-postgres/schemasee the column SQL type change; the TS types stayDate.How did you test your changes?
New
test/timezone.test.ts(testcontainerspostgres:15-alpine, skipped on Windows like the other container tests; per-database time zones are set withALTER DATABASE ... SET timezone):America/Los_AngelesandAsia/Kolkata, and a UTC server with a pool settingTimeZone=Asia/Tokyo: drive run/step/hook/wait creation through the storage API and assert everycreatedAt/startedAt(write results,runs.get,steps.get,hooks.get,events.list) is the current instant.workflowistimestamp with time zoneand thecreated_atdefaults staynow().main) only, write rows the way 5.0.x did (defaults plus JS ISO updates), run all migrations, read back through the drizzle schema. InAmerica/Los_Angeles,created_atand never-updatedupdated_atequal the transaction's realnow()(and acreated_atfrom January is moved by the standard-time offset, not today's), while JS-writtenstarted_at/updated_at/resume_at/snapshotcreated_atare unchanged. On a UTC server the same holds andpg_relation_filenodeof all 8 tables and their indexes is unchanged exceptworkflow_invocations_pending, whose predicate readsresponded_at(no table rewrite). With rows written in aTimeZone=Asia/Tokyosession,workflow.legacy_timezone=Asia/Tokyorepairs them (without it they stay 9 h off), set either in the connection'soptionsor withALTER DATABASE.runMigrationsin parallel from the 0025 state (both insert a 0027 row in the migrations table), and the 0027 SQL executed again on a converted database, leave the repaired values where one run put them. Without the lock-and-skip guard both fail by 25,200,000 ms.workflow_events,bootstrapfails with55P03within the 10 s timeout, and succeeds once that transaction ends.process.env.TZ = 'America/Los_Angeles',retryAfter/startedAt/createdAtread back as the instants written (with drizzle's stock readerretryAftercomes back 7 h late).idxmatches its file number and itswhenis later than the one before, so a duplicatewhenlike this branch's pre-merge 0026 fails CI.Results:
mainat ecaa047):test/timezone.test.ts13/13 and the whole package 388/388 onpostgres:15-alpine; typecheck and Biome clean on changed files;node scripts/check-changesets.mjsOK. Breaking the reader alone fails only the pre-migration reader test; restoring the duplicatewhenfails the journal test and every upgrade test.main's schema 8 of the 9 tests fail (off by 25,200,000 ms, 19,800,000 ms and 32,400,000 ms; the UTC no-rewrite test is a guard and passes on both). With this PR: 9/9 onpostgres:15-alpine,postgres:12-alpineandpostgres:18-alpine(the image thee2e-local-postgresjob uses), and 9/9 on Homebrew PostgreSQL 14.20 after merging the latestmain.main:storage.test.ts,retention.test.tsandrun-creation.test.ts185/185 on PostgreSQL 14.20;pnpm vitest run src71/71;pnpm buildandpnpm typecheckclean; Biome clean on changed files;node scripts/check-changesets.mjsOK.Unit Tests (ubuntu-latest)job failed twice on this branch with 330/330 tests passing and two unhandled57P01errors attributed totest/timezone.test.ts(the teardown race above, now handled in this file and reproduced in isolation withpg_terminate_backendon an idle pooled client).maincurrently fails the same job the same way fromtest/invoke.test.ts(e.g. on b5c1bc8), so the job can still go red from the older test files until they get the same listener; that is pre-existing and not touched here. On the latest head the job failed earlier, inpackages/core(fanout-dispatch.test.ts"publishes every queued branch ... in chunks of at most 16": a wall-clock bound, 270 ms against< 260), which this PR does not touch; it needs a rerun to show the world-postgres result.pnpm build && pnpm test, testcontainers): 330/330 on three runs and 329/330 on one. Each failure was a single pre-existing timing test that passes in isolation and is unrelated to this diff:storage.test.ts"should release a retained hook after its deadline" (writes a hostDate.now() - 1and expects the container'snow()to be past it, on the already-timestamptztoken_retention_until) andinvoke.test.ts"slow fallback" (3 s budget under parallel load).main's storage code on servers inAmerica/Los_Angeles,Asia/Kolkata,Europe/London,America/St_Johns(-02:30) and UTC, ran the migrations, then continued the same run with the new code: every run/step/hook/wait/event timestamp is the instant it was written, the event log'screatedAtstays monotonic across the upgrade, andmain's code reading the migrated database from a UTC session gets the same instants.Docs Preview
Open questions for reviewers
timestamptzor the keep-timestampfallback? The fallback: keeptimestamp, change the defaults totimezone('utc', now()), fix the rawnow()writes ininvocations.ts, and keep the same repair UPDATE. It stays compatible with old readers in any session zone but leaves the trap for any futurenow()write. This PR prefers the type change.patch, per AGENTS.md (patchfor bug fixes); the review suggestedminorgiven the data-rewriting migration.Related PRs and merge order
mainshipped its own0026_add_seq_in_band([world] In-band writer fence for world-local and world-postgres #4821) with the journalwhenthis branch's 0026 had, so this migration is now0027. Without the renumber, databases already at main's 0026 would skip the conversion silently.[world-postgres] Scope startup recovery to the World's jobPrefix) also adds0027_add_run_job_prefixwith the samewhenas this PR's 0027, and does not have main's 0026 yet. Whichever of the two merges second has to take0028and a laterwhen; the journal-order test in this PR fails on the clash.PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete🤖 Generated with Claude Code