Skip to content

feat(sdk): stop waiting for operation plugin hooks - #1067

Merged
ParidelPooya merged 8 commits into
mainfrom
feat/plugin-hooks-non-blocking
Oct 23, 2026
Merged

ParidelPooya merged 8 commits into
mainfrom
feat/plugin-hooks-non-blocking

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The SDK awaited every plugin hook. That had two consequences:

  • A slow hook delayed the next operation.
  • Some hooks are awaited while no operation is EXECUTING. Examples are onOperationStart of a step, and onOperationEnd of a failed step or a replayed wait. If another operation was waiting, for example a callback, a hook slower than the termination cooldown let the invocation suspend before the caller's next line ran. If that line submits the callback, the workflow waits until the callback times out.

Plugins are observers, so they should not decide when operations run. This PR stops waiting for operation hooks and drains their work once, at the end of the invocation.

Change

  • onOperationStart, onOperationEnd, onOperationAttemptStart, onOperationAttemptEnd and onOperationChange are called and not awaited.
    • The synchronous part of each hook still runs before the SDK continues. Hooks keep their call order, and state a hook sets synchronously is visible to later hooks. The OTel plugin relies on this: wrapOperationAttemptFn looks up the span that onOperationAttemptStart created.
    • The asynchronous parts of consecutive hooks can now overlap.
  • Before it calls onInvocationEnd, the runner waits for the pending hook work to settle, including work that settling work starts. It then waits for onInvocationEnd.
  • Two deadlines bound the waits. The drain stops 500 ms before the invocation's deadline (PLUGIN_DRAIN_RESERVE_MS). The wait for onInvocationEnd stops 50 ms before it (PLUGIN_END_RESERVE_MS), so slow operation-hook work cannot take the flush's time, and an invocation that ends close to its timeout still flushes. Work that has not settled by its deadline is abandoned. Before, a hook that never settled held the invocation until the Lambda timeout.
  • The runner now reads the remaining time on every invocation that has a plugin. A context that lacks getRemainingTimeInMillis, throws from it, or returns a non-finite value counts as having no deadline, in the execution context and in the runner. So a stub context in a unit test still works.
  • Unchanged: onInvocationStart is still awaited, because it runs before any operation and plugins use it for setup. The wrap hooks, providePropagationMetadata and enrichLogContext are unchanged.
  • The plugin docs in types/plugin.ts and the telemetry exporter's contract docs describe the new behavior.

Behavior changes for plugin authors

  • A hook can no longer rely on its asynchronous work finishing before the next operation runs. Work that must finish before the response belongs in onInvocationEnd, or is drained there automatically.
  • onInvocationEnd is now called after the drain, not synchronously when the invocation ends.
  • A plugin that needs the asynchronous parts of its hooks in order must queue them itself.

The shipped plugins already work this way. The OTel plugins do their operation-hook work synchronously and flush in onInvocationEnd. The telemetry exporter schedules exports on its own queue and drains it in onInvocationEnd. Their test suites pass unchanged.

This changes the documented contract of an @experimental interface. CHANGELOG.md records it under Unreleased as breaking for plugin authors; the commit-title lint does not accept feat(sdk)!:, so the release notes should list it under Breaking Changes. The Python and Java SDKs still await operation hooks.

Recording the rules

The second commit records the rules from this PR and #1065 so a later change cannot remove them unnoticed:

  • docs/adr/0001-operation-tracking-and-plugin-hooks.md states the five rules (the same numbering as the package AGENTS.md), why they exist, and the measured failures they prevent. Changing a rule needs a new ADR that supersedes it.
  • packages/aws-durable-execution-sdk-js/AGENTS.md lists the rules for people and coding agents that change the SDK, with the test that guards each. The root AGENTS.md is for customers, so the rules do not go there. The package publishes only dist*, so this file is not shipped.
  • CONTRIBUTING.md links to both.
  • .github/prompts/ai-pr-review.md tells the AI reviewer to read the package AGENTS.md from the base branch. It reports a broken rule, or an edited contract test without a superseding ADR, at the highest severity. The reviewer reads the prompt from the base branch, so this takes effect after merge.
  • .github/CODEOWNERS assigns the files that implement or guard the rules to @aws/aws-lambda-durable-functions-js-maintain. It blocks merges only once branch protection on main requires review from code owners. That setting needs a repository admin.
  • plugin-non-blocking.composed.test.ts is renamed to plugin-hook-contract.composed.test.ts. The three guarding test files state in their header that a failing case means a broken rule.

Not in this PR

All plugins share the drain. The runner calls onInvocationEnd on every plugin only after the pending work of all plugins has settled, or the drain deadline has passed. So when one plugin's operation hooks are slow, another plugin's flush gets at most 450 ms. A per-plugin drain would remove this coupling. The ADR records it under Consequences, and it can follow in a separate PR, together with an optional deadline or AbortSignal on InvocationEndInfo.

Testing

  • New composed test plugin-hook-contract.composed.test.ts, which runs the real engine with hooks slower than the cooldown:
    • The next step runs without waiting for four slow hooks on the previous step.
    • onInvocationEnd is called after the earlier hooks' work settled.
    • With a callback pending, the caller's next line runs before suspension when a failed step's onOperationEnd is slow, and when a step's onOperationStart is slow.
    • The engine bounds the drain by the context's remaining time, and still answers when the context cannot report it.
    • On main, cases 1, 3, 4 and 5 fail. Case 2 passes on both, since it pins behavior that must not change. Case 6 passes on both, because main never reads the remaining time on that path.
  • New runner unit tests: hooks are not awaited; synchronous parts run in call order; onInvocationStart is still awaited; the drain runs before onInvocationEnd; the drain covers work started during the drain; the drain and onInvocationEnd have separate deadlines; onInvocationEnd still flushes with less time left than the drain reserve; an unreadable, NaN or non-number remaining time waits for everything; timers are capped at MAX_TIMER_DELAY_MS. Each of these fails when the guard it covers is removed.
  • npm run build, npm test (all packages) and npm run lint pass. Core SDK: 1537 passed. Telemetry exporter: 159 passed. OTel: 791 passed. Testing library: 1444 passed, 3 skipped. Examples: 288 passed, 3 skipped.

…ocation end

The SDK awaited every plugin hook. A slow hook delayed the next
operation. A hook awaited while no operation was EXECUTING could also let
the invocation suspend before the caller's next line ran, when another
operation such as a callback was waiting.

The plugin runner now calls onOperationStart, onOperationEnd,
onOperationAttemptStart, onOperationAttemptEnd and onOperationChange and
returns without waiting. The synchronous part of each hook still runs
before the SDK continues, so hooks keep their call order and state they
set synchronously is visible to later wrap hooks. The runner keeps the
returned promises and, before it calls onInvocationEnd, waits for them
to settle. The drain and onInvocationEnd share a budget of the
invocation's remaining time minus 500 ms, so a hook that never settles
no longer holds the invocation until the Lambda timeout.

onInvocationStart is still awaited: it runs before any operation. The
wrap hooks, providePropagationMetadata and enrichLogContext are
unchanged.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 17:07 — with GitHub Actions Inactive
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 17:08 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

…uard them

ADR 0001 states four rules: an operation stays EXECUTING until its
result is ready, a child context is tracked from the moment its function
returns or throws, operation plugin hooks are never awaited, and hook
work is drained only before onInvocationEnd within the remaining time.
It records why, with the measured early-suspension failures.

The rules are now recorded where each audience reads them:
- packages/aws-durable-execution-sdk-js/AGENTS.md lists them for people
  and coding agents that change the SDK, with the tests that guard each.
- CONTRIBUTING.md links to both.
- The AI PR review prompt reads the package AGENTS.md from the base
  branch and reports a broken invariant, or an edited contract test
  without a superseding ADR, at the highest severity.
- CODEOWNERS requires a JS maintainer review for the files that
  implement or guard the rules.

The plugin composed test is renamed to plugin-hook-contract, and the
three guarding test files say in their header that a failing case means
a broken invariant.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 17:24 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 17:56 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

…eadline

Review fixes for the non-blocking plugin hooks:

- The runner read the remaining time on every invocation with a plugin,
  without a guard. A context that lacked getRemainingTimeInMillis, or
  threw from it, made the invocation throw, and no onInvocationEnd ran.
  The execution context and the runner now read a missing, throwing or
  non-finite value as no deadline.
- The 500 ms reserve applied to onInvocationEnd too. So an invocation
  that ended with under about 550 ms left skipped the flush. The drain
  of earlier hook work now stops 500 ms before the deadline, and the
  wait for onInvocationEnd stops 50 ms before it.
- settleBy passed delays above 2^31 - 1 ms to setTimeout, which Node
  runs after 1 ms. Each timer is now capped at MAX_TIMER_DELAY_MS.
- Composed tests now cover the deadline wiring in the engine and a
  context that cannot report its remaining time.
- Docs: the ADR and AGENTS.md use the same five rule numbers, the ADR
  gives the map overhead with its setup and range, plugin-contracts.md
  states the deadlines, and CHANGELOG records the plugin contract change
  as breaking for plugin authors.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 18:24 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

…e; record the shared drain

- A runner test now has a never-settling operation hook and a 100 ms
  flush with 700 ms left. It fails if the drain reserve equals the end
  reserve, because the hook would then use the flush's time.
- execution-context.test.ts gets a partial contract header naming the
  remaining-time cases, and the ADR's Enforcement section lists them.
- The ADR's Consequences record that all plugins share the drain, so one
  plugin's slow hooks limit another's flush to 450 ms, and that a
  per-plugin drain is not part of this decision. The map overhead reads
  5 to 10 %, matching its numbers.
- CHANGELOG states that a stub context made ctx.wait and step retries
  fail before, even without a plugin.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 18:46 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

…ejecting wrap hooks

Review feedback on the plugin contract:

- A child context was marked EXECUTING only after the whole
  wrapChildContextFn chain settled. A wrapper that awaited fn and then
  did more async work left no operation EXECUTING meanwhile, so with a
  callback waiting the invocation suspended before the caller's next
  line. The context is now marked EXECUTING as soon as the customer
  function settles.
- runAsCallback contained synchronous wrapper errors but not a wrapper
  promise that rejected on its own, for example after fn succeeded.
  That rejection escaped the Lambda handler. Such a rejection now falls
  back to the customer function's own result, and calls the function if
  the wrapper never did. An error from the customer function still wins.
- The plugin docs and the ADR now say the ordering guarantee is about
  when hooks are called (parent start, child start, child end, parent
  end), not when their async work finishes. A composed test pins the
  call order for nested operations and shows a slow child end finishing
  after the parent end.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 19:53 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

…ers still run

With plugins [A, B], an async rejection from A's wrapper before it
called next made the runner call the customer function directly, which
skipped B's wrapper. If B is the OTel plugin, the function then ran
without its span context.

Each link of the wrapper chain now contains its own wrapper's failure.
It reuses the inner result if the wrapper already called next, and
calls next if it did not. An error from the customer function still
wins. Two two-plugin tests cover a rejection before and after next.
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 20:10 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 20:31 — with GitHub Actions Inactive

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@github-actions

Copy link
Copy Markdown

Codex AI review

No actionable findings in the PR changes. The prior wrapper-chain issue is fixed. Residual test risk: this review was static; tests were not executed.

Reviewed commit 20aac688d5e5bf8cd369d6217ebec8e12007d9e8. Workflow run

@ParidelPooya
ParidelPooya merged commit 4e4e940 into main Oct 23, 2026
47 checks passed
@ParidelPooya
ParidelPooya deleted the feat/plugin-hooks-non-blocking branch October 23, 2026 21:02

This branch was previously deployed

1 inactive deployment
ai-pr-review-runtime — 20aac688 Deployed Oct 23, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #1745
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants