Repository navigation
fix(sdk): stop the invocation's work at its deadline and after termination - #1202
Merged
Merged
Conversation
The new tests start a step and a child context after the invocation suspended. Both run their function. They fail on this commit.
After termination the checkpoint manager accepts no checkpoint, but an operation that the handler started still ran. A step ran its function, because its START checkpoint is not awaited. On Lambda Managed Instances the handler keeps running after termination, so that step had side effects that the service never recorded. DurableContext now returns a promise that never settles when the invocation has terminated. The check sits where every operation starts. The test mocks of TerminationManager get the new method.
This comment has been minimized.
This comment has been minimized.
A step that waits for its signal is never aborted, so the invocation never answers. A step that ignores its signal is checkpointed after the deadline, and the next step runs. Both tests fail on this commit. The other two tests guard the no-deadline case and the timer cleanup.
On Lambda Managed Instances an invocation timeout does not stop the code that runs. The SDK read the deadline only to bound its polling, so after the deadline it kept checkpointing and starting operations. withDurableExecution now sets a timer for the remaining time. When it fires, it reads the remaining time again. If time is left, it sets the timer again. Otherwise the invocation terminates with INVOCATION_DEADLINE_PASSED. The existing termination path then aborts the signal of every running step, stops checkpoints, and refuses new operations. The invocation throws InvocationDeadlineError, an invocation error, so Lambda retries the invocation and the execution does not fail. No timer is set when the context reports no finite deadline. The timer is cleared when the invocation returns. The re-read keeps fake timers and the local test runner, whose context reports a constant 900000 ms, from ending an invocation early. Test fixtures: four execution context mocks get getRemainingTimeMs, and the invoke propagation test reported 0 ms left, which is now a passed deadline, so it reports 300000 ms.
…nces The guide said that the SDK does not check the deadline and does not abort the step signal at the deadline. Both are now false. The step example passes ctx.signal alone, without a second timeout signal.
This comment has been minimized.
This comment has been minimized.
Both Lambda contexts reported 0 ms left. The SDK now reads 0 ms as a passed deadline and ends the invocation, so the tests failed. They now report 300000 ms. No expectation changed.
This comment has been minimized.
This comment has been minimized.
A checkpoint queued behind a request in flight at the deadline is sent after termination. A step that an abort listener starts at the deadline runs. A finite remaining time above the setTimeout limit is passed to setTimeout as is. The three tests fail on this commit.
- processQueue drops queued updates once the invocation terminates. checkpoint() already refused new ones, but updates queued behind a batch in flight were still sent when that batch returned. - TerminationManager.terminate sets its flag before it runs the checkpoint callback. The callback aborts the invocation signal, so an abort listener that starts an operation now sees the termination. - The deadline timer caps each delay at MAX_TIMER_DELAY_MS. setTimeout turns a larger delay into 1 ms, so a long finite deadline made the timer fire and re-arm every millisecond. The no-deadline test now checks that no deadline timer is set, so it fails if the Number.isFinite guard is removed.
This comment has been minimized.
This comment has been minimized.
The previous commit made processQueue drop queued updates once the invocation terminates. A fault termination, such as a context validation error, needs those updates: the service records the failed execution with them. So three context-validation examples failed. The drop is removed, with its test. After the deadline, an update that was queued behind a request in flight can still be sent. On Lambda Managed Instances the service rejected such a late update with "Invalid checkpoint token" once the retry ran.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…-after-termination # Conflicts: # packages/aws-durable-execution-sdk-js/src/handlers/run-in-child-context-handler/run-in-child-context-composed.test.ts # packages/aws-durable-execution-sdk-js/src/handlers/run-in-child-context-handler/run-in-child-context-serdes.composed.test.ts # packages/aws-durable-execution-sdk-js/src/with-durable-execution-queue-completion.test.ts # packages/aws-durable-execution-sdk-js/src/with-durable-execution.plugin-lifetime.composed.test.ts
A step that ignores its signal and returns after the deadline still runs its serdes. A serdes can write the result somewhere, so the old invocation can overwrite what the retry stored. The test fails on this commit.
A step that returns after the deadline still ran its serdes before the refused checkpoint. A serdes can write the value somewhere, such as a file keyed by the step, so the old invocation could overwrite what the retry stored. safeSerialize now returns a promise that never settles once the invocation terminated. Every handler serializes through it. Test fixtures: - Hand-written TerminationManager mocks in the handler tests get isTerminationRequested. No expectation changed. - Two ReplayChildren fixtures in the run-in-child-context composed tests had no Name or SubType. Replay validation terminated them as non-deterministic, and they passed only because work went on after the termination. They now record the name and subtype the code uses.
This comment has been minimized.
This comment has been minimized.
8 tasks
This comment has been minimized.
This comment has been minimized.
Codex AI reviewTwo prior findings remain: checkpoint draining beyond the deadline ( Suppressed repeats
Reviewed commit |
Slow integration testsSlow means ≥1.5x the median + 1s, or ≥5s over the median. The median is from recent runs.
|
hln33
approved these changes
Nov 12, 2026
12 tasks
yaythomas
pushed a commit
that referenced
this pull request
Nov 13, 2026
## Issue Link Closes #1101 ## Description After an invocation returns, unfinished work could still submit checkpoints because `CheckpointManager.dispose()` did not close checkpoint admission. Disposal now reuses `setTerminating()` to refuse new requests, and `processQueue()` drops unsent queued work. Ordinary termination still drains its queue. Requests already in flight retain their completion behavior, including force-checkpoint waiters and accepted execution-result checkpoints. Invocation cleanup also aborts `StepContext.signal`, allowing steps the handler did not await to stop. The public signal documentation now includes normal return and error exits. `checkAndTerminate()` refuses to arm a termination timer after disposal. The deadline cleanup test introduced by #1202 now checks two independent outcomes: the signal is aborted when the invocation returns, and advancing beyond the original deadline does not call `terminate()` with `INVOCATION_DEADLINE_PASSED`. Fake `Date` and timers advance together; `setImmediate` remains real so checkpoint processing can drain. Runtime behavior is unchanged by this test correction. ## Design considerations - Reuse the existing admission gate rather than adding disposal checks to each checkpoint entry point. The dispatch guard remains necessary because normal termination must drain queued requests while disposal abandons them. This retains the lifecycle approach previously approved in this PR. - Do not clear the queue immediately in `dispose()`: that would discard force-checkpoint waiters attached to an active request. - The old deadline test used an unaborted signal as a proxy for timer cleanup. Completion now intentionally aborts that signal. Merely changing `false` to `true` would lose timer-cleanup coverage, so the test observes deadline termination directly. Removing completion abort would leave unfinished steps without their stop signal. ## Validation Fresh local checks for the deadline-test and documentation update: - Reproduced the original failure on `1f3ccda6`: `does not abort the signal after the invocation returned` failed with `expected true to be false` (1 failed, 7 filtered out). - Revised deadline and step-abort composed suites: 14 tests passed. - Temporarily removing `clearTimeout(deadlineTimer)` makes the revised test fail on an unexpected `INVOCATION_DEADLINE_PASSED` call. Independently removing the cleanup `terminated.abort()` makes it fail on the signal assertion. Both runtime lines were restored before final validation. - Complete SDK suite with coverage: 1,604 tests passed across 123 files on Node 22.23.4 and Node 24.21.0, including the SDK contract tests. Coverage thresholds passed. - SDK typecheck, lint (220 files), ESM/CommonJS/declaration builds, and `git diff --check` passed. The first build attempt lacked a shared Rollup plugin because only core dependencies were installed; installing the additional workspace dependencies from the lockfile resolved the local setup issue. The existing disposal tests cover refused admission, queued work abandoned before dispatch, in-flight completion, execution-result outcomes, termination draining, and absence of termination after disposal. The earlier admission regression evidence recorded in this PR showed two failures on pre-fix parent `48e75ca4`, then two passes with the admission fix. The existing OTel installed-consumer typecheck case has a 30-second timeout, matching its adjacent compiler test. In [the failed Node 22 job](https://github.com/aws/aws-durable-execution-sdk-js/actions/runs/43302215706/job/131481296020), compilation exceeded Vitest's default five-second limit. Its strict typechecking and assertions are preserved; all seven cases were verified on Node 22.23.4 in the prior timeout follow-up. That package was not rerun for this core test/documentation update. ## Scope and unverified behavior - An already in-flight checkpoint can finish after disposal. The separate polling-timer continuation in `forceRefreshAndCheckStatus()` remains outside this fix; subsequent checkpoint dispatch is blocked. - The existing deadline test's expectation changes because normal completion now aborts the signal. Its original timer-cleanup regression coverage is retained. No protected SDK invariant test is changed. - Cloud integration and conformance were not run locally. Live Lambda behavior remains unverified by these local checks. The cause of the earlier installed-consumer compilation slowdown is also unverified. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Pooya Paridel <ParidelPooya@users.noreply.github.com> Co-authored-by: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com>
7 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1102. Closes #1127 by documenting the behavior. Part of #927.
Problem
On Lambda Managed Instances (LMI), an invocation timeout does not stop the code that runs (LMI execution environment). Lambda retries the invocation. So the SDK keeps working for an invocation that Lambda has already answered, next to the retry of the same execution.
Cause
There are two gaps.
STARTcheckpoint is not awaited for the default AT_LEAST_ONCE semantics (step-handler.ts). So a step that the handler starts after termination runs its function at once. This holds for every termination reason, not only the deadline.Design
withDurableExecutionsets a timer forgetRemainingTimeMs(). When it fires, it reads the remaining time again. If time is left, it sets the timer again. Otherwise it callsterminate({ reason: INVOCATION_DEADLINE_PASSED }). The timer is cleared when the invocation returns. No timer is set when the context reports no finite deadline.terminate()already callssetTerminating()and then aborts the invocation'sAbortSignal. So every running step'sctx.signalaborts, and a step that stops on it cannot checkpoint aRETRYorFAIL.DurableContext.checkForNonResolvingPromisereturns a promise that never settles onceTerminationManager.isTerminationRequested()is true. Every operation starts through this method.InvocationDeadlineError, anUnrecoverableInvocationError. Lambda retries the invocation, and the execution does not fail.Why the timer reads the remaining time again: the context decides when the deadline is, not the timer. Fake timers can fire the timer early. The local test runner's context also reports a constant 900000 ms. Without the re-read,
skipTimetests in the testing package end invocations early.Why
InvocationDeadlineErrorand not PENDING or FAILED:Simpler options considered
LAMBDA_MANAGED_INSTANCES.md)abort()aloneRETRYor aFAIL. A timeout then becomes a step failure, or fails the execution when no retries are left.terminate()stops checkpoints first.map,parallel,runInChildContextandwaitForCallbackrun user functions that do not use the signal at all.STARTcheckpoint inCheckpointManagerSTARTis not awaited.Commits
test(sdk): show that an operation runs after the invocation terminatedfix(sdk): start no operation after the invocation terminatedtest(sdk): show that the invocation does not stop at its deadlinefix(sdk): terminate the invocation when its deadline passesdocs(sdk): describe the deadline termination for Lambda Managed InstancesStepContext.signalTSDoc.test(otel): report time left in two propagation test contextstest(sdk): show three gaps in the deadline terminationfix(sdk): close three gaps in the deadline terminationprocessQueuedrops queued updates after termination.terminate()sets its flag before the callback that aborts the signal. The timer caps each delay atMAX_TIMER_DELAY_MS.fix(sdk): send queued checkpoints after a termination againdocs(sdk): say that RETRYING does not promise a later invocationPluginInvocationStatus.RETRYINGTSDoc and the plugin LMI rules say that a stop or an execution timeout during a step can end the last invocation withRETRYING.docs(sdk): tell plugins to end open operations on RETRYINGonInvocationEnd, a plugin ends every operation that is still in progress or retrying, with its status ininfo.operations, and does not wait for a later invocation.test(sdk): show that a step result is serialized after the deadlinefix(sdk): start no serialization after the invocation terminatedsafeSerializereturns a promise that never settles after termination, so a serdes that writes files cannot overwrite what the retry stored. Handler test mocks getisTerminationRequested. Two ReplayChildren fixtures get theNameandSubTypethe code uses: without them replay validation terminated them as non-deterministic, and they passed only because work went on after the termination.#1127
On LMI, a stop or an execution timeout during a step leaves the invocation running. The service rejects its next checkpoint, and the SDK reports
RETRYINGtoonInvocationEnd, but no retry follows. The SDK cannot tell that rejection from one that Lambda retries, because both arrive as the same invocation error.onInvocationEndInvocationEndInfoUnverified: that the service rejects the checkpoint after a stop or an execution timeout comes from the #1127 report, not from a run in this PR.
Test output before the fixes:
Each branch has a test that fails without it. Without
Number.isFinite,setTimeout(Infinity)fires after 1 ms, and the no-deadline test fails. Without the re-read, the fake-timer test fails. WithoutclearTimeout, the cleanup test sees an aborted signal.Decisions not covered by the issue
The timer fires at the deadline, with no margin. A margin would end work that still has time, and the SDK cannot know how long the next step takes.
A refused operation gets a promise that never settles, like the other termination paths. It does not throw, because a throw would let the handler's
catchrun more code.The gate applies to every termination reason. Every termination ends the invocation, so no reason needs a later operation.
INVOCATION_DEADLINE_PASSEDis classed"fault"inTERMINATION_CLASS, but it is handled before that map, likeCHECKPOINT_FAILEDandSERDES_FAILED.Plugins see
onInvocationEndwith statusRETRYINGand theInvocationDeadlineError, like the other invocation errors.TerminationReason,TerminationManager.isTerminationRequested()andInvocationDeadlineErrorare internal and not exported.Edited test fixtures, with no expectation changed: four execution context mocks get
getRemainingTimeMs, and threeTerminationManagermocks getisTerminationRequested.invoke-propagation.composed.test.tsand two OTel propagation tests reported 0 ms left, which is now a passed deadline, so they report 300000 ms.After the deadline, the wrapper still waits for a checkpoint request in flight, and an update queued behind it is still sent. A fault termination needs both, and the deadline uses the same termination path. On LMI the service rejected such a late update with
Invalid checkpoint token. The default client times out a request after 55000 ms.Out of scope
context.promise.race([Bug]: LMI invocations can outlive deadlines and continue checkpointing after disposal #927).Verification
npx vitest run --coveragepasses 122 files and 1581 tests.npm run lintandtsc -p tsconfig.build.json --noEmitpass.skipTimeclock tests.extrasandotelpackages were not built. They are not related to this change.ctx.signal, or it rejects when the signal aborts. The same probe ran onmainand on this branch.mainInvalid checkpoint tokenCheckpointUnrecoverableInvocationError5 s after the deadlineInvocationDeadlineErrorat the deadlineaborted=falseaborted=true, result not checkpointedafterstep run by the old invocationafterran once (in the retry)afterran once (in the retry)The service already rejects a checkpoint from the timed-out invocation once the retry runs. So on
main, the damage is the late step work and one rejected checkpoint, not a secondafter. The runtime logs410 Invoke timeoutwhen the old invocation answers, so Lambda ignores that answer, as expected. TheInvocationDeadlineErrorwas logged 8 ms before the history'sInvocationCompletedevent. That is why the timer must not fire before the deadline.