Repository navigation
Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs #2978
Description
Summary
When multiple independent applications share one @workflow/world-postgres database, they execute each other's workflow runs. Queue namespaces (WORKFLOW_QUEUE_NAMESPACE) do not prevent this, for two independent reasons:
-
Claiming ignores namespaces. The graphile-worker task identifiers are
${jobPrefix}flows/${jobPrefix}steps— the namespace only prefixes the topic string inside the message. Any consumer polling the same database claims any namespace's jobs, andcreateTaskHandlerthen reconstructs the queue name using the consumer's own prefix (const queueName =${queue}${messageData.id}``), so the foreign job is executed locally instead of being skipped. -
Startup recovery is unscoped.
reenqueueActiveRunslists all pending/running runs with no namespace/deployment filter and re-enqueues each under the booting application's namespace prefix. So every application boot takes over every other application's in-flight runs. Fix namespaced active run recovery #2888 fixed which namespace the recovered runs are enqueued under, and [world-local] Scope untagged startup recovery to untagged runs #2667 scoped world-local recovery by tag — but the Postgres world still passes its raw, unfilteredstorage.runsinto recovery (packages/world-postgres'sstart()), so cross-application takeover remains. Notably,workflow_runs.deployment_idexists in the schema but world-postgres hardcodesgetDeploymentId()to'postgres', so it can't be used for scoping today.
Code references (checked against @workflow/world@5.0.0-beta.21 and @workflow/world-postgres@5.0.0-beta.27, the latest betas as of 2026-07-17):
packages/world/src/recovery.ts—runs.list({ status, resolveData: 'none', ... })with no scoping parameter.packages/world-postgres/src/queue.ts—getJobQueueName()derives the task identifier fromjobPrefixonly;createTaskHandler()rebuilds the queue name from the local prefix without validating the producer's namespace.packages/world-postgres/src/index.ts—start()→reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres', config.namespace).
How we hit this
We run a self-hosted platform that hosts many independent Eve agents (each is its own app/process), and initially pointed all of them at one WORKFLOW_POSTGRES_URL. Eve already derives a distinct WORKFLOW_QUEUE_NAMESPACE per agent, so topics were namespaced — yet agents intermittently executed each other's turns with the wrong application's code and credentials (surfacing as flaky provider/API-key errors, depending on which runner won the claim). Tracing it led to the two mechanisms above.
Repro sketch
- Two apps A and B (any two workflow apps with same-named workflows — e.g. two instances of the same framework), same
WORKFLOW_POSTGRES_URL, distinctWORKFLOW_QUEUE_NAMESPACE. - Start a long-running workflow in A.
- While it is
running, boot B. B logs[world-postgres] Re-enqueued N active run(s) on startupincluding A's run, under B's namespace. - B's runner claims and executes A's run (same
workflow_flowstask id; handler reconstructs the topic with B's prefix). With distinctWORKFLOW_POSTGRES_JOB_PREFIXvalues the takeover becomes deterministic: the re-enqueued foreign run can only be claimed by B.
Suggested directions
Either would resolve this class of issue:
- Scope claiming and recovery to the world's identity — e.g. include the namespace in the graphile task identifier (or validate the producer namespace in the handler and re-enqueue/skip on mismatch), and give
runs.lista filter thatreenqueueActiveRunspasses through (thedeployment_idcolumn looks like a natural hook, similar in spirit to the world-local tag scoping in [world-local] Scope untagged startup recovery to untagged runs #2667). - Or, if a Postgres world database is intended to be exclusive to a single application, state that explicitly in the world-postgres docs — today nothing fails loudly when two applications share one database; they just silently cross-execute.
Workaround
We moved to one Postgres database per application (derived from a base URL, created + workflow-postgres-setup bootstrapped on demand). That fully isolates storage, queue, and recovery, and has been working well — sharing it here for other self-hosters who hit the same symptom.
Activity
fix(recovery): address review — fail-safe NODE_ENV polarity, WORKFLOW…
hikaru-212 commentedon Sep 11, 2026
I dug into this locally and was able to reproduce both failure modes described here.
For the claim path, App A can create a run/job and App B can still claim it when both applications share the same Graphile task.
The job is then dispatched using B’s locally reconstructed queue namespace.
For startup recovery, starting App B can also cause
reenqueueActiveRuns()to pick up pending/running runs created by App A, because the current Postgres run storage has no application-level ownership filter.I added a local regression covering both cases.
Against the unpatched implementation I tested locally, both checks fail in the expected way:
I also built a local prototype to test one possible fix direction.
With that prototype, the same regression passes without changing the assertions, and the existing
@workflow/world-postgrestest suite still passes.What seems to be missing is a durable application identity carried consistently across storage, queueing, dispatch, and recovery.
In the prototype, I currently:
workflow_runsBefore turning this into a PR, I wanted to check what direction you would prefer for the ownership identity and migration shape.
Would a Postgres-specific
applicationIdbe reasonable here, or would you prefer to extend/reuse an existing concept such asnamespaceorjobPrefix?Persisting the owner on
workflow_runswould also require a schema migration, so I’d rather align on that before polishing the patch.For scope, this prototype is only addressing the cross-application isolation problem in #2978.
It is not intended to solve:
mechiland commentedon Sep 11, 2026
Thanks for digging in and reproducing both paths — that matches what we traced when we filed this.
A caveat up front: I'm a user, not a maintainer, so I can't tell you which direction they'd accept. What I can share is what we ended up doing, plus one thing I noticed re-reading
maintoday that might save you a migration.What we did. We couldn't wait on a fix, so we went two ways. Short term, one database per application, which is still what runs in production for us. Longer term we wrote our own World for the multi-tenant case (evelandhq/workflow-world), where a tenant id is persisted on every row and leads every query, and the embedded runner's Graphile task name is suffixed per tenant. One lesson from that which applies to your prototype: don't try to express "only claim my jobs" with Graphile flags.
forbiddenFlagsis a deny-list and can't say that. Partitioning the task identifier is what actually held for us.What's already in
main(as of 17bd649). This may be relevant to your "which identity" question:workflow_runs.deployment_idalready exists and is NOT NULL. Every existing row holds'postgres', becausepackages/world-postgres/src/queue.tshardcodesgetDeploymentId()to that string.guardDeploymentAffinityinpackages/core/src/runtime/deployment-guard.ts. It comparesrun.deploymentIdwithworld.getDeploymentId()before every delivery, re-routes a misrouted message to the pinned deployment with a bounded retry budget, and recordsrun_failedwithDEPLOYMENT_MISMATCHwhen the budget is spent. That looks like the fail-closed validation your prototype does by hand.capabilities.deploymentAffinity === true, which world-postgres doesn't declare, so it's inert there today.QueueOptions.deploymentIdexists at the interface level; world-postgres ignores it.So a possible shape with no schema change: make the deployment id configurable (
config.deploymentId/WORKFLOW_POSTGRES_DEPLOYMENT_ID), add adeploymentIdfilter toruns.list(#3707 is touching that param right now), pass it fromreenqueueActiveRuns, suffix the task identifier with it, honouropts.deploymentIdinqueue(), and declare the capability. Whether the maintainers prefer that over a separateapplicationIdcolumn I honestly don't know; it just seemed worth pointing out that the column and the guard already exist.Two things from the operator side. Not objections, just what I'd worry about running it:
'postgres'rows, would have been what I wanted as a user.hikaru-212 commentedon Sep 25, 2026
Thanks — the
deploymentIdsuggestion and the upgrade concerns were really helpful. I went back and checked the deployment-affinity path and its existing runtime tests in more detail.With
deploymentAffinityenabled, the contract treats a deployment ID as one concrete/immutable build. The runtime tests enforce that pin: a run persisted for deployment A but delivered under deployment B is rerouted toward A, and if rerouting is exhausted the run is failed withDEPLOYMENT_MISMATCH.That makes me still lean toward a separate, stable
applicationIdfor the cross-redeployment application boundary in #2978, rather than reusingdeploymentIdfor both concepts.I do agree with your upgrade concern, though. I’d change the rollout shape from what I originally prototyped:
Also agreed on terminology — I’ll use “application identity/scope” rather than “owner” here, to keep it separate from the live-process ownership discussed in #3138/#3758.
This is still just the direction I’m leaning after checking the current implementation/tests; maintainer feedback on the identity and migration shape would still be very welcome.
pranaygp commentedon Oct 13, 2026
A data point from after #4835, which shipped in
@workflow/world5.0.4 and@workflow/world-postgres5.0.4. That change makesreenqueueActiveRunssend each run back to the queue namespace it recorded when it was created (executionContext.queueNamespace), so this issue is now partly addressed:Different namespaces: addressed for new runs. A run that recorded its own namespace (a non-Workflow queue consumer, or an app whose
start()records one) is re-enqueued under its own prefix, not the booting app's. Runs created before the upgrade recorded nothing and still fall back to the booting app's default.Same namespace: still open on 5.0.4. Two apps that record the same namespace still take over each other's in-flight runs on boot. Examples are two deployments of one framework, or two apps built on the same queue consumer. We reproduced it with two apps of a durable agent harness on one Postgres database, each with its own
WORKFLOW_POSTGRES_JOB_PREFIX:The job prefix keeps the two apps' live jobs apart, but recovery doesn't use it: it lists every pending or running run in the shared runs table.
#4761 (approved, not merged yet) fixes exactly that recovery half for world-postgres. It stamps each run with its World's
jobPrefixand scopesstart()'s recovery tojob_prefix = <own prefix> OR job_prefix IS NULL. With it, a distinctWORKFLOW_POSTGRES_JOB_PREFIXper app keeps both live jobs and recovery apart, which covers the reproduction above. The claim half (apps sharing a database and a prefix) stays open, as does the stableapplicationIddirection discussed above. Until #4761 ships, we tell users to give each app its own database, or its own data directory with world-local.[world-postgres] Scope startup recovery to the World's jobPrefix (#4761)