Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/quickjs-slot-report-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

The QuickJS engine now reports its log position on writes and feeds its VM from the events a World returns on the response, matching the node:vm engine.
27 changes: 27 additions & 0 deletions docs/content/docs/v5/how-it-works/event-sourcing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,33 @@ Both kinds of skip are logged at `debug`, so neither reaches the console unless

The observability UI grays out events it can identify this way and shows the reason on hover. Its set is narrower than the runtime's because it reads the log without consumer state. A consumer for an entity that is still open can legitimately claim a repeat because each step retry writes another `step_started`. The UI marks a repeat only when no consumer can remain for it: after a terminal event for the same entity or at a second `run_started`, of which the log records one per run. The UI marks nothing on a partial log view, such as one page of a paginated list or search results, because identifying the first copy requires the entire log.

## Events returned on a write

Concurrent invocations of one run write to a shared log, and a write is decided against the log the writer had loaded. When the write lands above other events the writer had not seen, the World can hand those events back on the write's own success response, so the writer merges them and continues rather than discovering the gap on its next read. Two parameters of `events.create()` ask for this:

- **`eventCount`** says how many events the writer held. When the write commits at a higher position than `eventCount + 1`, the World returns the events on the positions in between (the *skipped-slot report*).
- **`sinceCursor`** is the `events.list()` cursor the writer had read to. The World returns everything after it, the write itself included (the *inline delta*), in the same `events` / `cursor` / `hasMore` shape as a list page.

Both save a reload, so the rule for which writes send them is not about the event type. It is about whether the process that writes **holds a loaded log it will keep deciding from**: its own next writes, or the replay it resumes after the write. A writer with no log has nothing to merge a page into, and whoever decides next is a fresh replay that loads the log anyway. Three things must hold for a page to come back: the writer named a position, the World reads a page for that event type (a backend may decline for types whose writer never holds a log), and there is something to return (the report only when positions were skipped; the delta always).

| Event | Writer | Sends | Why |
|-------|--------|-------|-----|
| `run_created` | `start()` | Nothing | No log exists yet. |
| `run_started` | Replay, first write of a delivery | `eventCount` when a log is loaded; no cursor | The `run_started` response already carries the full log as a preload. |
| `step_created`, `wait_created`, `hook_disposed`, `attr_set` | Suspension handler | `eventCount` | The handler keeps writing from this log and the replay resumes from it. |
| `hook_created` | Suspension handler | `eventCount` and `sinceCursor` (only when the suspension creates exactly one hook) | The delta is how a hook whose payload already arrived resolves without another invocation. Two creates diffing against one cursor would give two deltas of which only the first could be taken. |
| `hook_received` | Replay resuming a hook | `eventCount` | The replay continues from the log right after. |
| `hook_received` | Webhook or `resumeHook()` | Nothing | Out-of-band. It enqueues a delivery that loads the log. |
| `wait_completed` | Replay completing an elapsed wait | `eventCount` | The replay continues from the log right after. |
| `step_started`, `step_retrying` | Step executor | Nothing | The executor holds no log. The next decision is a replay that reloads, or receives the delta on the step's terminal write. |
| `step_completed`, `step_failed` | Step executor | `sinceCursor` when a single step runs inline; nothing from a queued delivery | Inline, this is where the invocation gets its log back without a list. From a queue, a fresh replay loads the log anyway. |
| `attr_set` | `setAttributes()` outside a workflow | Nothing | Out-of-band. |
| `run_completed`, `run_failed`, `run_cancelled` | Replay, or `cancel()` | Never `sinceCursor` | Terminal. Nothing replays the log afterwards, so a page would be read by no one. |

Batched writes (`events.createBatch()`) carry no position and get no page. Both runtime engines follow this table: the `node:vm` engine merges a returned page into the log it replays from, and the QuickJS engine queues it for delivery to its live VM, in position order, ahead of its next `events.list()`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Both runtime engines follow this table" is not quite true for the hook_created row: the node suspension handler sends eventCount and sinceCursor on one hook write per suspension (that delta is how a hook whose payload already arrived resolves in-process), while this PR's QuickJS dispatchPendingOps sends only eventCount on hook_created. Either note the QuickJS exception in the row, or add the sinceCursor ask on the QuickJS side in a follow-up. The same applies to the helpers.ts copy of the table.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI) Added the sinceCursor ask on the QuickJS side in ebb5337 rather than noting an exception. dispatchPendingOps takes a deltaCursor from the loop and passes it as sinceCursor on the hook_created create when exactly one hook needs creating (the node handler's hookDeltaCursor gate); the createEvent seam takes a delta response through absorbDelta so the cursor moves with it. Pinned by a new test in quickjs-slot-report.test.ts (hook_created reaches the VM off the delta, no listing). Both table copies updated to say the ask is gated on exactly one hook, since that is the part that was imprecise for the node engine too.


A World that supports this returns the page in `events`, `cursor`, and `hasMore` on the `EventResult`. A World that does not simply returns the created event, and the runtime lists. See [Building a World](/worlds/building-a-world#event-id-allocation) for the World-side contract.

## Sealed positions (noop events)

Runs at `specVersion` 7 and above use a *sealed log*. The backend gives each write its position from a per-run sequencer **before** the write commits, so concurrent writers hold distinct positions and never race for a slot. New runs use this behavior by default. [`WORKFLOW_SEALED_LOG=0`](/docs/configuration/runtime-tuning#workflow_sealed_log) returns a deployment to the previous scheme. Every runtime reads a sealed log regardless, and a run's version is fixed at creation, so changing the setting never affects an in-flight run. However, a writer can claim a position and then stop because of a crashed process or canceled transaction. This leaves a hole that no writer will fill, and a hole looks like an event the reader failed to load.
Expand Down
2 changes: 2 additions & 0 deletions docs/content/worlds/v5/building-a-world.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,8 @@ The version a World stamps comes from `mintedSpecVersion()`: 7 by default or the

`events.create()` params carry `eventCount`: how many events the writer held in the log it replayed from, which is the position it expects to land on minus one. Attempt `eventCount + 1`. When that position is taken, **do not reject the write**. Advance to the next free position, commit there, and return the events occupying the positions you skipped over on the success response, in `events` with a matching `cursor` and `hasMore`. The writer merges them into its own log and replays once, rather than paying a second round trip to discover it was behind. A caller with a stale count is the normal case for a fan-out, and rejecting it would serialize writes the runtime issues in parallel.

Not every write carries `eventCount`, and a World is free to skip the report read for event types whose writer never holds a log to merge it into (the step executor's `step_started`, `step_retrying`, `step_completed`, and `step_failed`, and the run-terminal `run_completed`, `run_failed`, and `run_cancelled`). Which writes send `eventCount` or `sinceCursor`, and why, is tabulated in [Events returned on a write](/docs/how-it-works/event-sourcing#events-returned-on-a-write). When a write carries both, the `sinceCursor` delta is a superset of the report, so return the delta and skip the report.

### Optional: rejecting a stale write

A replay writes events derived from the event log it loaded, so a write made from an event log that no longer matches the store can commit events no correct replay would produce. Rejecting one means answering `events.create()` with a `PreconditionFailedError` when the run's log already holds more events than the caller's `eventCount` says it had loaded.
Expand Down
7 changes: 7 additions & 0 deletions packages/core/src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2885,6 +2885,13 @@ export function workflowEntrypoint(
? eventLog.events
: undefined,
preloadedEventsComplete: eventLog.type === 'ready',
// Where that log was read to, so the engine reads
// forward from it and can ask for an inline delta
// against it.
preloadedCursor:
eventLog.type === 'ready'
? eventLog.cursor
: undefined,
runInput,
parentSpan: span,
maxEventsLimit,
Expand Down
47 changes: 47 additions & 0 deletions packages/core/src/runtime/helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1032,6 +1032,53 @@ export interface SlotSnapshotParams {
* Empty for an empty log, which is the state a `run_created` write is issued
* from: there is no position held yet to name.
*
* ## Who names a position
*
* A World can hand events back on a write's success response through two
* params (`CreateEventParams` in `@workflow/world`): `eventCount`, which is
* answered with the events on the positions the write skipped over (the
* skipped-slot report), and `sinceCursor`, which is answered with everything
* after that cursor (the inline delta). Both exist to save the writer a
* reload. So the rule for which writes send them is not about the event type;
* it is about whether **the process that writes holds a loaded log it will
* keep deciding from**, its own next writes or the replay it resumes. A writer
* with no log has nothing to merge a page into, and whoever decides next is a
* fresh replay that loads the log anyway.
*
* Three gates, all of which must pass for a page to come back:
*
* 1. The writer names a position. Only the replay loop's `createEvent` seam
* (runtime.ts) and the suspension handler's `createGuarded` do; the QuickJS
* engine's `createEvent` (quickjs-entrypoint.ts) follows the same rule. Raw
* `world.events.create` calls send nothing.
* 2. The World reads a page for that event type. A World may decline for types
* whose writer never holds a log (step executor writes, run-terminal
* writes), whatever the client sent.
* 3. There is something to hand back: the report only when the write landed
* more than one position above `eventCount`; the delta always.
*
* | Event | Writer | Sends | Why |
* |---|---|---|---|
* | `run_created` | `start()` | nothing | no log exists yet |
* | `run_started` | replay loop, first write of a delivery | `eventCount` when a log is loaded; no cursor, the `run_started` preload owns the response fields | the preload already returns the full log |
* | `step_created`, `wait_created`, `hook_disposed`, `attr_set` | suspension handler (`createGuarded`) | `eventCount` | the handler keeps writing from, and resumes replay from, this log |
* | `step_created` | queue-consumer re-ensure (raw) | nothing | runs from a queue message, no log |
* | `hook_created` | suspension handler (node), `dispatchPendingOps` (QuickJS) | `eventCount` + `sinceCursor` (only when the suspension creates exactly one hook) | the delta is how an already-received hook resolves without a re-invocation; two creates against one cursor would give two deltas of which only the first could be taken |
* | `hook_received` | replay loop lazy resume; handler in-suspension resume | `eventCount` (preload owns the cursor fields) | both replay from the log right after |
* | `hook_received` | webhook / `resumeHook` (raw) | nothing | out-of-band, enqueues a delivery that loads the log |
* | `wait_completed` | replay loop timer path | `eventCount` | the loop keeps replaying from the log |
* | `wait_completed` | `runs.*` out-of-band (raw) | nothing | no log |
* | `step_started`, `step_retrying` | step executor | nothing | no log; the next decision is a replay that reloads or receives the delta below |
* | `step_completed`, `step_failed` | step executor | `sinceCursor` only, outside turbo, single inline step | this is where the inline execution loop gets its log back; in turbo or from a queued delivery a fresh replay loads it anyway |
* | `step_failed` | suspension handler, unserializable input | `eventCount` | the handler holds a log; rare, and follows a `step_created` that drew the page |
* | `attr_set` | `setAttributes()` API (raw) | nothing | out-of-band |
* | `run_completed`, `run_failed`, `run_cancelled` | replay loop, replay budget, `runs.cancel` | `eventCount` where the node loop's seam stamps it (the QuickJS engine sends nothing); never `sinceCursor` (`deltaRequestCursor` excludes terminal types) | terminal: nothing replays the log afterwards, so a World reads no page for them |
*
* Batched writes (`events.createBatch`) are outside all of this: a batch carries
* no per-event position and gets no page. The user-facing version of this table
* is in `docs/content/docs/v5/how-it-works/event-sourcing.mdx`; keep the two in
* step.
*
* The maximum rather than the length, for the reason {@link maxEventSlot}
* gives: a partially-read log holds fewer events than its highest position, and
* counting those would make the write claim to have seen less than it has, so
Expand Down
Loading
Loading