Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/enforce-strict-concurrency.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
"@workflow/sveltekit": patch
---

Add opt-in `WORKFLOW_SEQUENTIAL_REPLAYS` env var (also enabled by the `WORKFLOW_SAFE_MODE=1` umbrella flag when not set explicitly). When set to `1`, flow (orchestrator) routes are limited to one invocation per run at a time via a per-run queue topic and `maxConcurrency: 1` on the flow trigger. Step routes are unaffected.
Add opt-in `WORKFLOW_SEQUENTIAL_REPLAYS` env var. When set to `1`, flow (orchestrator) routes are limited to one invocation per run at a time via a per-run queue topic and `maxConcurrency: 1` on the flow trigger. Step routes are unaffected. Routing each run through a dedicated `maxConcurrency: 1` topic might lead to higher queue performance overhead.
6 changes: 6 additions & 0 deletions .changeset/remove-safe-mode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@workflow/world-vercel": patch
"@workflow/builders": patch
---

`WORKFLOW_SEQUENTIAL_REPLAYS` is configured only by its own environment variable.
11 changes: 3 additions & 8 deletions docs/content/docs/v4/deploying/world/vercel-world.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -122,21 +122,16 @@ Custom base URL for the Vercel workflow API. Automatically detected.

### `WORKFLOW_SEQUENTIAL_REPLAYS`

Set `WORKFLOW_SEQUENTIAL_REPLAYS=1` to guarantee that **at most one orchestrator (flow) invocation runs at a time per workflow run**. This behavior is off by default; without it, the runtime relies on idempotency and the event log to tolerate concurrent flow invocations of the same run. It is also enabled by `WORKFLOW_SAFE_MODE=1` when `WORKFLOW_SEQUENTIAL_REPLAYS` is not set explicitly.
Set `WORKFLOW_SEQUENTIAL_REPLAYS=1` to guarantee that **at most one orchestrator (flow) invocation runs at a time per workflow run**. This behavior is off by default; without it, the runtime relies on idempotency and the event log to tolerate concurrent flow invocations of the same run.

When enabled, each run is given its own queue topic and the flow route is configured with `maxConcurrency: 1`, so [Vercel Queues](https://vercel.com/docs/queues) processes flow messages for a given run strictly one at a time. Step routes are unaffected and continue to run with full concurrency.

<Callout type="warn">
This variable is read at **both build time and runtime**, so it must be set as a project-level environment variable that applies to your build and your deployed functions. Setting it for only one will produce an inconsistent configuration. The same applies to framework integrations that write their own queue trigger configuration instead of using `getWorkflowQueueTrigger()` from `@workflow/builders`: they only get the runtime half (per-run topics) unless they also emit `maxConcurrency: 1` on their flow trigger.

Enabling sequential replays has a cost. Per [Vercel Queues pricing](https://vercel.com/docs/queues/pricing), push deliveries under `maxConcurrency` are billed at **2x units** for that operation, so every flow-route delivery costs double while this is enabled. It also creates one queue topic per run, which increases the number of distinct queues surfaced in queue observability, and each flow invocation waits for a per-run concurrency slot before delivery, which can add queueing latency. Leave it off unless you specifically need the per-run serialization guarantee.


While a replay holds a run's slot — including time spent executing steps inline — other wake messages for that run (hook resumes, aborts and cancellations, and run-timeout enforcement) wait for the slot. Expect aborts and timeouts to be delayed by up to the duration of the longest single invocation.

The guarantee covers messages sent by the Workflow SDK itself. External producers that compute a flow topic name directly (rather than enqueueing through the SDK) still deliver, but bypass the per-run serialization slot.
</Callout>

Because it routes each run's flow invocations through a dedicated `maxConcurrency: 1` queue topic, enabling this might lead to higher queue performance overhead.

### Programmatic configuration

{/*@skip-typecheck: incomplete code sample*/}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@ const flowTriggers = [getWorkflowQueueTrigger()];
const stepTriggers = [STEP_QUEUE_TRIGGER];
```

If your integration constructs the flow trigger object itself instead of calling `getWorkflowQueueTrigger()`, it must add `maxConcurrency: 1` to that trigger when sequential replays are enabled at build time (`WORKFLOW_SEQUENTIAL_REPLAYS=1`, or `WORKFLOW_SAFE_MODE=1` when the specific variable is unset — the exported `isSequentialReplaysEnabled()` helper implements this check). The runtime half of the feature (per-run queue topics) activates from the environment variable alone — without the trigger half, those per-run topics are not serialized and the setting only adds queue-topic cardinality.
If your integration constructs the flow trigger object itself instead of calling `getWorkflowQueueTrigger()`, it must add `maxConcurrency: 1` to that trigger when sequential replays are enabled at build time (`WORKFLOW_SEQUENTIAL_REPLAYS=1` — the exported `isSequentialReplaysEnabled()` helper implements this check). The runtime half of the feature (per-run queue topics) activates from the environment variable alone — without the trigger half, those per-run topics are not serialized and the setting only adds queue-topic cardinality.


### Custom implementations
Expand Down
3 changes: 1 addition & 2 deletions docs/content/docs/v5/configuration/worlds.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -201,8 +201,7 @@ Platform-provided values such as `VERCEL_DEPLOYMENT_ID`, `VERCEL_PROJECT_ID`, an
- Default: disabled
- Set `1` to serialize orchestrator (flow) invocations per run: each run's replays get their own queue topic and the flow trigger is generated with `maxConcurrency: 1`. Inline step executions get per-step topics and keep full parallelism.
- Read at **both build time and runtime** — set it as a project-level environment variable so the generated trigger and the runtime queue routing agree.
- Also enabled by `WORKFLOW_SAFE_MODE=1` when not set explicitly.
- Costs: flow-route push deliveries under `maxConcurrency` are billed at 2x units, queue observability sees one topic per run, and deliveries can queue behind the per-run slot — including hook resumes, aborts, and run-timeout enforcement, which are delayed while a replay is in flight. See [Vercel World](/docs/deploying/world/vercel-world#workflow_sequential_replays) for details.
- Routing each run through a dedicated `maxConcurrency: 1` topic might lead to higher queue performance overhead. See [Vercel World](/docs/deploying/world/vercel-world#workflow_sequential_replays) for details.

### `VERCEL_WORKFLOW_SERVER_URL`

Expand Down
11 changes: 3 additions & 8 deletions docs/content/docs/v5/deploying/world/vercel-world.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -171,21 +171,16 @@ Custom workflow-server URL for direct runtime requests, or for the proxy to forw

### `WORKFLOW_SEQUENTIAL_REPLAYS`

Set `WORKFLOW_SEQUENTIAL_REPLAYS=1` to guarantee that **at most one orchestrator (flow) invocation runs at a time per workflow run**. This behavior is off by default; without it, the runtime relies on idempotency and the event log to tolerate concurrent flow invocations of the same run. It is also enabled by `WORKFLOW_SAFE_MODE=1` when `WORKFLOW_SEQUENTIAL_REPLAYS` is not set explicitly.
Set `WORKFLOW_SEQUENTIAL_REPLAYS=1` to guarantee that **at most one orchestrator (flow) invocation runs at a time per workflow run**. This behavior is off by default; without it, the runtime relies on idempotency and the event log to tolerate concurrent flow invocations of the same run.

When enabled, each run's orchestrator messages are given their own queue topic and the flow trigger is configured with `maxConcurrency: 1`, so [Vercel Queues](https://vercel.com/docs/queues) processes replays for a given run strictly one at a time. Step executions (which ride the flow topic in the combined handler model) get a per-step topic, so steps keep full parallelism.

<Callout type="warn">
This variable is read at **both build time and runtime**, so it must be set as a project-level environment variable that applies to your build and your deployed functions. Setting it for only one will produce an inconsistent configuration. The same applies to framework integrations that write their own queue trigger configuration instead of using `getWorkflowQueueTrigger()` from `@workflow/builders`: they only get the runtime half (per-run topics) unless they also emit `maxConcurrency: 1` on their flow trigger.

Enabling sequential replays has a cost. Per [Vercel Queues pricing](https://vercel.com/docs/queues/pricing), push deliveries under `maxConcurrency` are billed at **2x units** for that operation, so every flow-route delivery costs double while this is enabled. It also creates one queue topic per run, which increases the number of distinct queues surfaced in queue observability, and each flow invocation waits for a per-run concurrency slot before delivery, which can add queueing latency. Leave it off unless you specifically need the per-run serialization guarantee.


While a replay holds a run's slot — including time spent executing steps inline — other wake messages for that run (hook resumes, aborts and cancellations, and run-timeout enforcement) wait for the slot. Expect aborts and timeouts to be delayed by up to the duration of the longest single invocation.

The guarantee covers messages sent by the Workflow SDK itself. External producers that compute a flow topic name directly (rather than enqueueing through the SDK) still deliver, but bypass the per-run serialization slot.
</Callout>

Because it routes each run's flow invocations through a dedicated `maxConcurrency: 1` queue topic, enabling this might lead to higher queue performance overhead.

### `VERCEL_QUEUE_MAX_DELAY_SECONDS`

Maximum delay, in seconds, that Workflow uses for one Vercel Queues continuation message when implementing `sleep()`. If a workflow sleeps longer than this, the runtime schedules another continuation message when the first one fires, repeating until the sleep's target time is reached. Default: `82800` (23 hours).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@ const flowTriggers = [getWorkflowQueueTrigger()];
const stepTriggers = [STEP_QUEUE_TRIGGER];
```

If your integration constructs the flow trigger object itself instead of calling `getWorkflowQueueTrigger()`, it must add `maxConcurrency: 1` to that trigger when sequential replays are enabled at build time (`WORKFLOW_SEQUENTIAL_REPLAYS=1`, or `WORKFLOW_SAFE_MODE=1` when the specific variable is unset — the exported `isSequentialReplaysEnabled()` helper implements this check). The runtime half of the feature (per-run queue topics) activates from the environment variable alone — without the trigger half, those per-run topics are not serialized and the setting only adds queue-topic cardinality.
If your integration constructs the flow trigger object itself instead of calling `getWorkflowQueueTrigger()`, it must add `maxConcurrency: 1` to that trigger when sequential replays are enabled at build time (`WORKFLOW_SEQUENTIAL_REPLAYS=1` — the exported `isSequentialReplaysEnabled()` helper implements this check). The runtime half of the feature (per-run queue topics) activates from the environment variable alone — without the trigger half, those per-run topics are not serialized and the setting only adds queue-topic cardinality.


### Custom implementations
Expand Down
20 changes: 0 additions & 20 deletions packages/builders/src/constants.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,9 @@ import {

describe('getWorkflowQueueTrigger', () => {
let originalStrict: string | undefined;
let originalSafeMode: string | undefined;

beforeEach(() => {
originalStrict = process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
originalSafeMode = process.env.WORKFLOW_SAFE_MODE;
delete process.env.WORKFLOW_SAFE_MODE;
});

afterEach(() => {
Expand All @@ -22,11 +19,6 @@ describe('getWorkflowQueueTrigger', () => {
} else {
delete process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
}
if (originalSafeMode !== undefined) {
process.env.WORKFLOW_SAFE_MODE = originalSafeMode;
} else {
delete process.env.WORKFLOW_SAFE_MODE;
}
});

it('omits maxConcurrency by default', () => {
Expand All @@ -51,18 +43,6 @@ describe('getWorkflowQueueTrigger', () => {
expect('maxConcurrency' in trigger).toBe(false);
});

it('WORKFLOW_SAFE_MODE=1 sets maxConcurrency when the specific variable is unset', () => {
delete process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
process.env.WORKFLOW_SAFE_MODE = '1';
expect(getWorkflowQueueTrigger()).toMatchObject({ maxConcurrency: 1 });
});

it('an explicit WORKFLOW_SEQUENTIAL_REPLAYS=0 wins over WORKFLOW_SAFE_MODE', () => {
process.env.WORKFLOW_SEQUENTIAL_REPLAYS = '0';
process.env.WORKFLOW_SAFE_MODE = '1';
expect('maxConcurrency' in getWorkflowQueueTrigger()).toBe(false);
});

it('composes with an explicit namespace option', () => {
process.env.WORKFLOW_SEQUENTIAL_REPLAYS = '1';
expect(getWorkflowQueueTrigger({ namespace: 'custom' })).toMatchObject({
Expand Down
12 changes: 3 additions & 9 deletions packages/builders/src/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -118,17 +118,11 @@ export const WORKFLOW_QUEUE_TRIGGER = createWorkflowQueueTrigger();
* the route's `experimentalTriggers` config.
*/
/**
* Whether sequential replays are enabled: `WORKFLOW_SEQUENTIAL_REPLAYS=1`,
* or `WORKFLOW_SAFE_MODE=1` when `WORKFLOW_SEQUENTIAL_REPLAYS` is not set
* explicitly (safe mode fills the default of every safety-over-performance
* flag; an explicit per-flag value always wins). Read at call time.
* Whether sequential replays are enabled (`WORKFLOW_SEQUENTIAL_REPLAYS=1`). Read
* at call time.
*/
export function isSequentialReplaysEnabled(): boolean {
const explicit = process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
if (explicit !== undefined && explicit !== '') {
return explicit === '1';
}
return process.env.WORKFLOW_SAFE_MODE === '1';
return process.env.WORKFLOW_SEQUENTIAL_REPLAYS === '1';
}

export function getWorkflowQueueTrigger(options?: { namespace?: string }) {
Expand Down
28 changes: 0 additions & 28 deletions packages/world-vercel/src/queue.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -397,13 +397,10 @@ describe('createQueue', () => {
describe('strict concurrency (WORKFLOW_SEQUENTIAL_REPLAYS)', () => {
let originalDeploymentId: string | undefined;
let originalStrict: string | undefined;
let originalSafeMode: string | undefined;

beforeEach(() => {
originalDeploymentId = process.env.VERCEL_DEPLOYMENT_ID;
originalStrict = process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
originalSafeMode = process.env.WORKFLOW_SAFE_MODE;
delete process.env.WORKFLOW_SAFE_MODE;
process.env.VERCEL_DEPLOYMENT_ID = 'dpl_test';
mockSend.mockResolvedValue({ messageId: 'msg-123' });
});
Expand All @@ -419,11 +416,6 @@ describe('createQueue', () => {
} else {
delete process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
}
if (originalSafeMode !== undefined) {
process.env.WORKFLOW_SAFE_MODE = originalSafeMode;
} else {
delete process.env.WORKFLOW_SAFE_MODE;
}
});

it('appends runId to the physical flow topic while keeping the logical queueName', async () => {
Expand Down Expand Up @@ -491,26 +483,6 @@ describe('createQueue', () => {
expect(mockSend.mock.calls[0][0]).toBe('__wkf_step_myStep');
});

it('WORKFLOW_SAFE_MODE=1 routes to per-run topics when the specific variable is unset', async () => {
delete process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
process.env.WORKFLOW_SAFE_MODE = '1';

const queue = createQueue();
await queue.queue('__wkf_workflow_test', { runId: 'wrun_abc' });

expect(mockSend.mock.calls[0][0]).toBe('__wkf_workflow_test_wrun_abc');
});

it('an explicit WORKFLOW_SEQUENTIAL_REPLAYS=0 wins over WORKFLOW_SAFE_MODE', async () => {
process.env.WORKFLOW_SEQUENTIAL_REPLAYS = '0';
process.env.WORKFLOW_SAFE_MODE = '1';

const queue = createQueue();
await queue.queue('__wkf_workflow_test', { runId: 'wrun_abc' });

expect(mockSend.mock.calls[0][0]).toBe('__wkf_workflow_test');
});

it('gives inline step executions (flow topic + stepId) a per-step topic for full parallelism', async () => {
process.env.WORKFLOW_SEQUENTIAL_REPLAYS = '1';

Expand Down
15 changes: 4 additions & 11 deletions packages/world-vercel/src/queue.ts
Original file line number Diff line number Diff line change
Expand Up @@ -294,19 +294,12 @@ const FLOW_TOPIC_PATTERN = /^__([a-z][a-z0-9]*_)?wkf_workflow_/;
let loggedSequentialReplays = false;

/**
* Whether sequential replays are enabled: `WORKFLOW_SEQUENTIAL_REPLAYS=1`,
* or `WORKFLOW_SAFE_MODE=1` when `WORKFLOW_SEQUENTIAL_REPLAYS` is not set
* explicitly (safe mode fills the default of every safety-over-performance
* flag; an explicit per-flag value always wins). Mirrors
* `isSequentialReplaysEnabled` in `@workflow/builders` — world-vercel must
* not depend on the build-time package, so the check is duplicated.
* Whether sequential replays are enabled (`WORKFLOW_SEQUENTIAL_REPLAYS=1`).
* Mirrors `isSequentialReplaysEnabled` in `@workflow/builders` — world-vercel
* must not depend on the build-time package, so the check is duplicated.
*/
function isSequentialReplaysEnabled(): boolean {
const explicit = process.env.WORKFLOW_SEQUENTIAL_REPLAYS;
if (explicit !== undefined && explicit !== '') {
return explicit === '1';
}
return process.env.WORKFLOW_SAFE_MODE === '1';
return process.env.WORKFLOW_SEQUENTIAL_REPLAYS === '1';
}

function getPhysicalQueueName(
Expand Down
Loading