Repository navigation
feat(sdk): stop waiting for operation plugin hooks - #1067
Merged
Merged
Conversation
…ocation end The SDK awaited every plugin hook. A slow hook delayed the next operation. A hook awaited while no operation was EXECUTING could also let the invocation suspend before the caller's next line ran, when another operation such as a callback was waiting. The plugin runner now calls onOperationStart, onOperationEnd, onOperationAttemptStart, onOperationAttemptEnd and onOperationChange and returns without waiting. The synchronous part of each hook still runs before the SDK continues, so hooks keep their call order and state they set synchronously is visible to later wrap hooks. The runner keeps the returned promises and, before it calls onInvocationEnd, waits for them to settle. The drain and onInvocationEnd share a budget of the invocation's remaining time minus 500 ms, so a hook that never settles no longer holds the invocation until the Lambda timeout. onInvocationStart is still awaited: it runs before any operation. The wrap hooks, providePropagationMetadata and enrichLogContext are unchanged.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 17:07 — with
GitHub Actions
Inactive
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 17:08 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
…uard them ADR 0001 states four rules: an operation stays EXECUTING until its result is ready, a child context is tracked from the moment its function returns or throws, operation plugin hooks are never awaited, and hook work is drained only before onInvocationEnd within the remaining time. It records why, with the measured early-suspension failures. The rules are now recorded where each audience reads them: - packages/aws-durable-execution-sdk-js/AGENTS.md lists them for people and coding agents that change the SDK, with the tests that guard each. - CONTRIBUTING.md links to both. - The AI PR review prompt reads the package AGENTS.md from the base branch and reports a broken invariant, or an edited contract test without a superseding ADR, at the highest severity. - CODEOWNERS requires a JS maintainer review for the files that implement or guard the rules. The plugin composed test is renamed to plugin-hook-contract, and the three guarding test files say in their header that a failing case means a broken invariant.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 17:24 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 17:56 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
…eadline Review fixes for the non-blocking plugin hooks: - The runner read the remaining time on every invocation with a plugin, without a guard. A context that lacked getRemainingTimeInMillis, or threw from it, made the invocation throw, and no onInvocationEnd ran. The execution context and the runner now read a missing, throwing or non-finite value as no deadline. - The 500 ms reserve applied to onInvocationEnd too. So an invocation that ended with under about 550 ms left skipped the flush. The drain of earlier hook work now stops 500 ms before the deadline, and the wait for onInvocationEnd stops 50 ms before it. - settleBy passed delays above 2^31 - 1 ms to setTimeout, which Node runs after 1 ms. Each timer is now capped at MAX_TIMER_DELAY_MS. - Composed tests now cover the deadline wiring in the engine and a context that cannot report its remaining time. - Docs: the ADR and AGENTS.md use the same five rule numbers, the ADR gives the map overhead with its setup and range, plugin-contracts.md states the deadlines, and CHANGELOG records the plugin contract change as breaking for plugin authors.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 18:24 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
…e; record the shared drain - A runner test now has a never-settling operation hook and a 100 ms flush with 700 ms left. It fails if the drain reserve equals the end reserve, because the hook would then use the flush's time. - execution-context.test.ts gets a partial contract header naming the remaining-time cases, and the ADR's Enforcement section lists them. - The ADR's Consequences record that all plugins share the drain, so one plugin's slow hooks limit another's flush to 450 ms, and that a per-plugin drain is not part of this decision. The map overhead reads 5 to 10 %, matching its numbers. - CHANGELOG states that a stub context made ctx.wait and step retries fail before, even without a plugin.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 18:46 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
yaythomas
reviewed
Oct 23, 2026
yaythomas
reviewed
Oct 23, 2026
yaythomas
reviewed
Oct 23, 2026
…ejecting wrap hooks Review feedback on the plugin contract: - A child context was marked EXECUTING only after the whole wrapChildContextFn chain settled. A wrapper that awaited fn and then did more async work left no operation EXECUTING meanwhile, so with a callback waiting the invocation suspended before the caller's next line. The context is now marked EXECUTING as soon as the customer function settles. - runAsCallback contained synchronous wrapper errors but not a wrapper promise that rejected on its own, for example after fn succeeded. That rejection escaped the Lambda handler. Such a rejection now falls back to the customer function's own result, and calls the function if the wrapper never did. An error from the customer function still wins. - The plugin docs and the ADR now say the ordering guarantee is about when hooks are called (parent start, child start, child end, parent end), not when their async work finishes. A composed test pins the call order for nested operations and shows a slow child end finishing after the parent end.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 19:53 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
…ers still run With plugins [A, B], an async rejection from A's wrapper before it called next made the runner call the customer function directly, which skipped B's wrapper. If B is the OTel plugin, the function then ran without its span context. Each link of the wrapper chain now contains its own wrapper's failure. It reuses the inner result if the wrapper already called next, and calls next if it did not. An error from the customer function still wins. Two two-plugin tests cover a rejection before and after next.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 20:10 — with
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 20:31 — with
GitHub Actions
Inactive
Codex AI reviewNo actionable findings in the PR changes. The prior wrapper-chain issue is fixed. Residual test risk: this review was static; tests were not executed. Reviewed commit |
This was referenced Oct 25, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The SDK awaited every plugin hook. That had two consequences:
EXECUTING. Examples areonOperationStartof a step, andonOperationEndof a failed step or a replayedwait. If another operation was waiting, for example a callback, a hook slower than the termination cooldown let the invocation suspend before the caller's next line ran. If that line submits the callback, the workflow waits until the callback times out.Plugins are observers, so they should not decide when operations run. This PR stops waiting for operation hooks and drains their work once, at the end of the invocation.
Change
onOperationStart,onOperationEnd,onOperationAttemptStart,onOperationAttemptEndandonOperationChangeare called and not awaited.wrapOperationAttemptFnlooks up the span thatonOperationAttemptStartcreated.onInvocationEnd, the runner waits for the pending hook work to settle, including work that settling work starts. It then waits foronInvocationEnd.PLUGIN_DRAIN_RESERVE_MS). The wait foronInvocationEndstops 50 ms before it (PLUGIN_END_RESERVE_MS), so slow operation-hook work cannot take the flush's time, and an invocation that ends close to its timeout still flushes. Work that has not settled by its deadline is abandoned. Before, a hook that never settled held the invocation until the Lambda timeout.getRemainingTimeInMillis, throws from it, or returns a non-finite value counts as having no deadline, in the execution context and in the runner. So a stub context in a unit test still works.onInvocationStartis still awaited, because it runs before any operation and plugins use it for setup. Thewraphooks,providePropagationMetadataandenrichLogContextare unchanged.types/plugin.tsand the telemetry exporter's contract docs describe the new behavior.Behavior changes for plugin authors
onInvocationEnd, or is drained there automatically.onInvocationEndis now called after the drain, not synchronously when the invocation ends.The shipped plugins already work this way. The OTel plugins do their operation-hook work synchronously and flush in
onInvocationEnd. The telemetry exporter schedules exports on its own queue and drains it inonInvocationEnd. Their test suites pass unchanged.This changes the documented contract of an
@experimentalinterface.CHANGELOG.mdrecords it under Unreleased as breaking for plugin authors; the commit-title lint does not acceptfeat(sdk)!:, so the release notes should list it under Breaking Changes. The Python and Java SDKs still await operation hooks.Recording the rules
The second commit records the rules from this PR and #1065 so a later change cannot remove them unnoticed:
docs/adr/0001-operation-tracking-and-plugin-hooks.mdstates the five rules (the same numbering as the packageAGENTS.md), why they exist, and the measured failures they prevent. Changing a rule needs a new ADR that supersedes it.packages/aws-durable-execution-sdk-js/AGENTS.mdlists the rules for people and coding agents that change the SDK, with the test that guards each. The rootAGENTS.mdis for customers, so the rules do not go there. The package publishes onlydist*, so this file is not shipped.CONTRIBUTING.mdlinks to both..github/prompts/ai-pr-review.mdtells the AI reviewer to read the packageAGENTS.mdfrom the base branch. It reports a broken rule, or an edited contract test without a superseding ADR, at the highest severity. The reviewer reads the prompt from the base branch, so this takes effect after merge..github/CODEOWNERSassigns the files that implement or guard the rules to@aws/aws-lambda-durable-functions-js-maintain. It blocks merges only once branch protection onmainrequires review from code owners. That setting needs a repository admin.plugin-non-blocking.composed.test.tsis renamed toplugin-hook-contract.composed.test.ts. The three guarding test files state in their header that a failing case means a broken rule.Not in this PR
All plugins share the drain. The runner calls
onInvocationEndon every plugin only after the pending work of all plugins has settled, or the drain deadline has passed. So when one plugin's operation hooks are slow, another plugin's flush gets at most 450 ms. A per-plugin drain would remove this coupling. The ADR records it under Consequences, and it can follow in a separate PR, together with an optionaldeadlineorAbortSignalonInvocationEndInfo.Testing
plugin-hook-contract.composed.test.ts, which runs the real engine with hooks slower than the cooldown:onInvocationEndis called after the earlier hooks' work settled.onOperationEndis slow, and when a step'sonOperationStartis slow.main, cases 1, 3, 4 and 5 fail. Case 2 passes on both, since it pins behavior that must not change. Case 6 passes on both, becausemainnever reads the remaining time on that path.onInvocationStartis still awaited; the drain runs beforeonInvocationEnd; the drain covers work started during the drain; the drain andonInvocationEndhave separate deadlines;onInvocationEndstill flushes with less time left than the drain reserve; an unreadable, NaN or non-number remaining time waits for everything; timers are capped atMAX_TIMER_DELAY_MS. Each of these fails when the guard it covers is removed.npm run build,npm test(all packages) andnpm run lintpass. Core SDK: 1537 passed. Telemetry exporter: 159 passed. OTel: 791 passed. Testing library: 1444 passed, 3 skipped. Examples: 288 passed, 3 skipped.