Skip to content

fix(sdk): prevent checkpoint requests after manager disposal - #1114

Merged
yaythomas merged 6 commits into
aws:mainfrom
itsrayeedscodespace:fix/issue-1101
Nov 13, 2026
Merged

yaythomas merged 6 commits into
aws:mainfrom
itsrayeedscodespace:fix/issue-1101

Conversation

@itsrayeedscodespace

@itsrayeedscodespace itsrayeedscodespace commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Issue Link

Closes #1101

Description

After an invocation returns, unfinished work could still submit checkpoints because CheckpointManager.dispose() did not close checkpoint admission. Disposal now reuses setTerminating() to refuse new requests, and processQueue() drops unsent queued work. Ordinary termination still drains its queue. Requests already in flight retain their completion behavior, including force-checkpoint waiters and accepted execution-result checkpoints.

Invocation cleanup also aborts StepContext.signal, allowing steps the handler did not await to stop. The public signal documentation now includes normal return and error exits. checkAndTerminate() refuses to arm a termination timer after disposal.

The deadline cleanup test introduced by #1202 now checks two independent outcomes: the signal is aborted when the invocation returns, and advancing beyond the original deadline does not call terminate() with INVOCATION_DEADLINE_PASSED. Fake Date and timers advance together; setImmediate remains real so checkpoint processing can drain. Runtime behavior is unchanged by this test correction.

Design considerations

  • Reuse the existing admission gate rather than adding disposal checks to each checkpoint entry point. The dispatch guard remains necessary because normal termination must drain queued requests while disposal abandons them. This retains the lifecycle approach previously approved in this PR.
  • Do not clear the queue immediately in dispose(): that would discard force-checkpoint waiters attached to an active request.
  • The old deadline test used an unaborted signal as a proxy for timer cleanup. Completion now intentionally aborts that signal. Merely changing false to true would lose timer-cleanup coverage, so the test observes deadline termination directly. Removing completion abort would leave unfinished steps without their stop signal.

Validation

Fresh local checks for the deadline-test and documentation update:

  • Reproduced the original failure on 1f3ccda6: does not abort the signal after the invocation returned failed with expected true to be false (1 failed, 7 filtered out).
  • Revised deadline and step-abort composed suites: 14 tests passed.
  • Temporarily removing clearTimeout(deadlineTimer) makes the revised test fail on an unexpected INVOCATION_DEADLINE_PASSED call. Independently removing the cleanup terminated.abort() makes it fail on the signal assertion. Both runtime lines were restored before final validation.
  • Complete SDK suite with coverage: 1,604 tests passed across 123 files on Node 22.23.4 and Node 24.21.0, including the SDK contract tests. Coverage thresholds passed.
  • SDK typecheck, lint (220 files), ESM/CommonJS/declaration builds, and git diff --check passed. The first build attempt lacked a shared Rollup plugin because only core dependencies were installed; installing the additional workspace dependencies from the lockfile resolved the local setup issue.

The existing disposal tests cover refused admission, queued work abandoned before dispatch, in-flight completion, execution-result outcomes, termination draining, and absence of termination after disposal. The earlier admission regression evidence recorded in this PR showed two failures on pre-fix parent 48e75ca4, then two passes with the admission fix.

The existing OTel installed-consumer typecheck case has a 30-second timeout, matching its adjacent compiler test. In the failed Node 22 job, compilation exceeded Vitest's default five-second limit. Its strict typechecking and assertions are preserved; all seven cases were verified on Node 22.23.4 in the prior timeout follow-up. That package was not rerun for this core test/documentation update.

Scope and unverified behavior

  • An already in-flight checkpoint can finish after disposal. The separate polling-timer continuation in forceRefreshAndCheckStatus() remains outside this fix; subsequent checkpoint dispatch is blocked.
  • The existing deadline test's expectation changes because normal completion now aborts the signal. Its original timer-cleanup regression coverage is retained. No protected SDK invariant test is changed.
  • Cloud integration and conformance were not run locally. Live Lambda behavior remains unverified by these local checks. The cause of the earlier installed-consumer compilation slowdown is also unverified.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@itsrayeedscodespace
itsrayeedscodespace requested a review from a team as a code owner October 27, 2026 19:34
@github-actions

This comment has been minimized.

@hln33

hln33 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

/ai review

@github-actions

Copy link
Copy Markdown

Codex AI review

No actionable findings. The previous rule 2 concern is addressed; SDK invariants and protected contract tests remain intact. Review was static only: tests and builds were not executed, so runtime race behavior remains unverified.

Reviewed commit 46c48f74ad610678bbb82361a6614854da12ece8. Workflow run

@itsrayeedscodespace

Copy link
Copy Markdown
Contributor Author

@zhongkechen could you please review this

@itsrayeedscodespace

Copy link
Copy Markdown
Contributor Author

@zhongkechen any updates/comments?

@itsrayeedscodespace

Copy link
Copy Markdown
Contributor Author

Follow-up pushed: 4b16e2cc1bf45edf57ba44e06c46b82ec8a0134d (normal fast-forward preserving the maintainer rebase). The five remaining Jest mock/spy calls now use Vitest; the direct review reply covers the correction.

The PR description records an executed pre-fix run on 48e75ca48522a4ddaa9cdf2f8509f2fef199de59: both post-disposal admission tests failed (scheduledCalls: expected 0, received 1), and the identical tests pass on this head. Final local validation passed: full root npm test, 209 checkpoint tests, all 1,572 SDK tests on Node 22 and 24, typecheck, lint/format, full build, and declared-files verification for all eight publishable packages.

Current Build and conformance workflows require maintainer approval (action_required, no Build jobs executed). The earlier external-PR gate separately requires the maintainer-controlled safe-to-merge label. No security gates were changed; cloud execution remains unverified locally. The reviewer-request API returned HTTP 404 for this account, so I could not add a formal review request. Please re-review when convenient; the threads remain open for the maintainers.

yaythomas
yaythomas previously approved these changes Nov 7, 2026

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ready to merge, pending CI.

thank you so much @itsrayeedscodespace, great contribution!

@itsrayeedscodespace

Copy link
Copy Markdown
Contributor Author

Thanks, @yaythomas! I really enjoyed working on this and appreciate your guidance through the review. I’d love to keep contributing—if there are any issues you’d like me to take on, or areas you’re looking to enhance or improve, please feel free to assign them to me. I’m happy to help with follow-up work too!

@itsrayeedscodespace

itsrayeedscodespace commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

@ParidelPooya thanks for the follow up fixes and review.

I see that the earlier approval was dismissed after the latest update, so it looks like a fresh review is also needed. Please let me know if there's anything you'd like me to address, or whether the workflow failure needs a maintainer-side rerun or permissions adjustment. Happy to help.

zhongkechen

This comment was marked as outdated.

SilanHe
SilanHe previously approved these changes Nov 12, 2026
@yaythomas
yaythomas force-pushed the fix/issue-1101 branch 2 times, most recently from 8502b8d to 054b480 Compare November 12, 2026 20:37
@zhongkechen
zhongkechen dismissed stale reviews from SilanHe and themself via 1f3ccda November 12, 2026 22:12
SilanHe
SilanHe previously approved these changes Nov 12, 2026
itsrayeedscodespace and others added 6 commits November 12, 2026 16:40
Two gaps after dispose():

- A normal return disposes the checkpoint manager but does not run the
  termination callback. So StepContext.signal was never aborted, and a
  step the handler did not await was never told to stop. Its next
  checkpoint now never settles, so the signal is its only stop. The
  finally block now aborts the signal after dispose().
- A checkpoint in flight at dispose() runs checkAndTerminate() when it
  completes. That armed a termination timer, which called terminate() on
  the ended invocation. checkAndTerminate() now returns when disposed.

Each new test fails with its fix reverted.
Give the strict installed-consumer typecheck a 30-second timeout, matching the adjacent packaged-consumer compiler test. The Node 22 CI case exceeded the default 5-second limit while compiling TypeScript.
Check completion abort separately from deadline timer cleanup, using a clock that advances with the timer. Update the public signal contract and cleanup comments to match disposal behavior.
@yaythomas
yaythomas merged commit 580cfa4 into aws:main Nov 13, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: CheckpointManager sends checkpoint requests after dispose()

6 participants