Skip to content

fix(sdk): stop the invocation's work at its deadline and after termination - #1202

Merged
ParidelPooya merged 19 commits into
mainfrom
fix/sdk-no-operations-after-termination
Nov 12, 2026
Merged

ParidelPooya merged 19 commits into
mainfrom
fix/sdk-no-operations-after-termination

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1102. Closes #1127 by documenting the behavior. Part of #927.

Problem

On Lambda Managed Instances (LMI), an invocation timeout does not stop the code that runs (LMI execution environment). Lambda retries the invocation. So the SDK keeps working for an invocation that Lambda has already answered, next to the retry of the same execution.

Cause

There are two gaps.

  1. The SDK reads the deadline only to bound its polling. It never ends the invocation at the deadline. So a running step is not aborted, its result is checkpointed, and the next step starts.
  2. Termination stops checkpoints, but it does not stop new operations. A step's START checkpoint is not awaited for the default AT_LEAST_ONCE semantics (step-handler.ts). So a step that the handler starts after termination runs its function at once. This holds for every termination reason, not only the deadline.

Design

Part Change
Deadline timer withDurableExecution sets a timer for getRemainingTimeMs(). When it fires, it reads the remaining time again. If time is left, it sets the timer again. Otherwise it calls terminate({ reason: INVOCATION_DEADLINE_PASSED }). The timer is cleared when the invocation returns. No timer is set when the context reports no finite deadline.
Existing termination path terminate() already calls setTerminating() and then aborts the invocation's AbortSignal. So every running step's ctx.signal aborts, and a step that stops on it cannot checkpoint a RETRY or FAIL.
Gate DurableContext.checkForNonResolvingPromise returns a promise that never settles once TerminationManager.isTerminationRequested() is true. Every operation starts through this method.
Answer The invocation throws InvocationDeadlineError, an UnrecoverableInvocationError. Lambda retries the invocation, and the execution does not fail.

Why the timer reads the remaining time again: the context decides when the deadline is, not the timer. Fake timers can fire the timer early. The local test runner's context also reports a constant 900000 ms. Without the re-read, skipTime tests in the testing package end invocations early.

Why InvocationDeadlineError and not PENDING or FAILED:

Answer Problem
PENDING The timer can fire just before Lambda's own timeout. Then the answer counts. A running step is not a pending operation, and the service rejects PENDING without one.
FAILED It fails the execution. A timeout should retry the invocation.
Invocation error It is the same outcome as Lambda's own timeout. When Lambda has already timed out, the answer is ignored.

Simpler options considered

Option Why it does not work
Docs only (LAMBDA_MANAGED_INSTANCES.md) The guide already warned about it. The SDK still checkpoints and starts steps after the deadline.
Check the clock where an operation starts It stops only new operations. A running step is not aborted, and its result is still checkpointed.
Timer that calls abort() alone The aborted step throws, and the step handler checkpoints a RETRY or a FAIL. A timeout then becomes a step failure, or fails the execution when no retries are left. terminate() stops checkpoints first.
Timer without the gate Code between steps has no signal. The next step's function runs with an already aborted signal, so it is safe only if its first line checks the signal. map, parallel, runInChildContext and waitForCallback run user functions that do not use the signal at all.
Refuse the START checkpoint in CheckpointManager It is already refused after termination. The step function runs anyway, because the START is not awaited.
A check in each handler About 10 copies of one check.

Commits

Commit Change
test(sdk): show that an operation runs after the invocation terminated A callback waits. The other branch sleeps outside any operation until the invocation suspends, and then starts a step or a child context. Both tests fail: the function runs.
fix(sdk): start no operation after the invocation terminated The gate. Both tests pass.
test(sdk): show that the invocation does not stop at its deadline A step that waits for its signal is never aborted, so the invocation never answers. A step that ignores its signal is checkpointed after the deadline, and the next step runs. Both tests fail. Two more tests guard the no-deadline case and the timer cleanup.
fix(sdk): terminate the invocation when its deadline passes The timer and the answer, plus a test for the re-read. All five tests pass.
docs(sdk): describe the deadline termination for Lambda Managed Instances Rules 1, 2 and 4 of the LMI guide, and the StepContext.signal TSDoc.
test(otel): report time left in two propagation test contexts Two OTel test contexts reported 0 ms left, which is now a passed deadline. They report 300000 ms.
test(sdk): show three gaps in the deadline termination A checkpoint queued behind a request in flight is sent after termination. A step started by an abort listener runs. A delay above the setTimeout limit is passed as is. All three fail.
fix(sdk): close three gaps in the deadline termination processQueue drops queued updates after termination. terminate() sets its flag before the callback that aborts the signal. The timer caps each delay at MAX_TIMER_DELAY_MS.
fix(sdk): send queued checkpoints after a termination again Removes the queue drop and its test. A fault termination, such as a context validation error, needs its queued updates sent. Three context-validation examples failed without them.
docs(sdk): say that RETRYING does not promise a later invocation #1127. The PluginInvocationStatus.RETRYING TSDoc and the plugin LMI rules say that a stop or an execution timeout during a step can end the last invocation with RETRYING.
docs(sdk): tell plugins to end open operations on RETRYING #1127. In onInvocationEnd, a plugin ends every operation that is still in progress or retrying, with its status in info.operations, and does not wait for a later invocation.
test(sdk): show that a step result is serialized after the deadline A step that ignores its signal returns after the deadline, and its serdes still runs. Fails.
fix(sdk): start no serialization after the invocation terminated safeSerialize returns a promise that never settles after termination, so a serdes that writes files cannot overwrite what the retry stored. Handler test mocks get isTerminationRequested. Two ReplayChildren fixtures get the Name and SubType the code uses: without them replay validation terminated them as non-deterministic, and they passed only because work went on after the termination.

#1127

On LMI, a stop or an execution timeout during a step leaves the invocation running. The service rejects its next checkpoint, and the SDK reports RETRYING to onInvocationEnd, but no retry follows. The SDK cannot tell that rejection from one that Lambda retries, because both arrive as the same invocation error.

Option Why not chosen
Skip onInvocationEnd The SDK cannot tell the two cases apart, so normal retries would lose the hook too. The hook is where plugins end spans and flush exporters, so the invocation's whole trace would be lost, not only its status.
Look up the real status and add it to InvocationEndInfo A public API change that needs a design note, for a case that only affects traces.
Document it Chosen. No behavior change.

Unverified: that the service rejects the checkpoint after a stop or an execution timeout comes from the #1127 report, not from a run in this PR.

Test output before the fixes:

× step: does not run the step function
× runInChildContext: does not run the child function
AssertionError: expected true to be false   // expect(ran).toBe(false)

× aborts a running step, records no failure for it, and fails the invocation
AssertionError: expected 'hung' to be an instance of Error
× does not record or follow a step that ignores the signal
AssertionError: expected undefined to be 'InvocationDeadlineError'

Each branch has a test that fails without it. Without Number.isFinite, setTimeout(Infinity) fires after 1 ms, and the no-deadline test fails. Without the re-read, the fake-timer test fails. Without clearTimeout, the cleanup test sees an aborted signal.

Decisions not covered by the issue

  • The timer fires at the deadline, with no margin. A margin would end work that still has time, and the SDK cannot know how long the next step takes.

  • A refused operation gets a promise that never settles, like the other termination paths. It does not throw, because a throw would let the handler's catch run more code.

  • The gate applies to every termination reason. Every termination ends the invocation, so no reason needs a later operation.

  • INVOCATION_DEADLINE_PASSED is classed "fault" in TERMINATION_CLASS, but it is handled before that map, like CHECKPOINT_FAILED and SERDES_FAILED.

  • Plugins see onInvocationEnd with status RETRYING and the InvocationDeadlineError, like the other invocation errors.

  • TerminationReason, TerminationManager.isTerminationRequested() and InvocationDeadlineError are internal and not exported.

  • Edited test fixtures, with no expectation changed: four execution context mocks get getRemainingTimeMs, and three TerminationManager mocks get isTerminationRequested. invoke-propagation.composed.test.ts and two OTel propagation tests reported 0 ms left, which is now a passed deadline, so they report 300000 ms.

  • After the deadline, the wrapper still waits for a checkpoint request in flight, and an update queued behind it is still sent. A fault termination needs both, and the deadline uses the same termination path. On LMI the service rejected such a late update with Invalid checkpoint token. The default client times out a request after 55000 ms.

Out of scope

Verification

  • SDK package: npx vitest run --coverage passes 122 files and 1581 tests. npm run lint and tsc -p tsconfig.build.json --noEmit pass.
  • Testing package, against the built SDK: 92 files and 1450 tests pass, including the skipTime clock tests.
  • Examples package: 270 tests pass. 5 files fail to load on this machine because the extras and otel packages were not built. They are not related to this change.
  • Lambda Managed Instances, nodejs24.x on arm64. A probe function has a 10 s Lambda timeout and a 15 s first step. The retry runs an 8 s step, so the old invocation's late step ends while the retry still runs. Two modes: the step ignores ctx.signal, or it rejects when the signal aborts. The same probe ran on main and on this branch.
Observation main This branch
Listening step aborted 5.2 s after the deadline, after a failed checkpoint 21 ms after the deadline
Late checkpoint from the old invocation Sent, rejected with Invalid checkpoint token Not sent
Old invocation's answer CheckpointUnrecoverableInvocationError 5 s after the deadline InvocationDeadlineError at the deadline
Step that ignores the signal Runs to the end, aborted=false Runs to the end, aborted=true, result not checkpointed
after step run by the old invocation No No
Execution result SUCCEEDED, after ran once (in the retry) SUCCEEDED, after ran once (in the retry)

The service already rejects a checkpoint from the timed-out invocation once the retry runs. So on main, the damage is the late step work and one rejected checkpoint, not a second after. The runtime logs 410 Invoke timeout when the old invocation answers, so Lambda ignores that answer, as expected. The InvocationDeadlineError was logged 8 ms before the history's InvocationCompleted event. That is why the timer must not fire before the deadline.

The new tests start a step and a child context after the invocation
suspended. Both run their function. They fail on this commit.
After termination the checkpoint manager accepts no checkpoint, but an
operation that the handler started still ran. A step ran its function,
because its START checkpoint is not awaited. On Lambda Managed Instances
the handler keeps running after termination, so that step had side
effects that the service never recorded.

DurableContext now returns a promise that never settles when the
invocation has terminated. The check sits where every operation starts.
The test mocks of TerminationManager get the new method.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 9, 2026 21:11 — with GitHub Actions Active
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 9, 2026 21:11 — with GitHub Actions Active
@github-actions

This comment has been minimized.

A step that waits for its signal is never aborted, so the invocation
never answers. A step that ignores its signal is checkpointed after the
deadline, and the next step runs. Both tests fail on this commit. The
other two tests guard the no-deadline case and the timer cleanup.
On Lambda Managed Instances an invocation timeout does not stop the
code that runs. The SDK read the deadline only to bound its polling, so
after the deadline it kept checkpointing and starting operations.

withDurableExecution now sets a timer for the remaining time. When it
fires, it reads the remaining time again. If time is left, it sets the
timer again. Otherwise the invocation terminates with
INVOCATION_DEADLINE_PASSED. The existing termination path then aborts
the signal of every running step, stops checkpoints, and refuses new
operations. The invocation throws InvocationDeadlineError, an
invocation error, so Lambda retries the invocation and the execution
does not fail. No timer is set when the context reports no finite
deadline. The timer is cleared when the invocation returns.

The re-read keeps fake timers and the local test runner, whose context
reports a constant 900000 ms, from ending an invocation early.

Test fixtures: four execution context mocks get getRemainingTimeMs, and
the invoke propagation test reported 0 ms left, which is now a passed
deadline, so it reports 300000 ms.
…nces

The guide said that the SDK does not check the deadline and does not
abort the step signal at the deadline. Both are now false. The step
example passes ctx.signal alone, without a second timeout signal.
@ParidelPooya ParidelPooya changed the title fix(sdk): start no operation after the invocation terminated fix(sdk): stop the invocation's work at its deadline and after termination Nov 9, 2026
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 9, 2026 22:48 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Both Lambda contexts reported 0 ms left. The SDK now reads 0 ms as a
passed deadline and ends the invocation, so the tests failed. They now
report 300000 ms. No expectation changed.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 10, 2026 00:12 — with GitHub Actions Active
@github-actions

This comment has been minimized.

A checkpoint queued behind a request in flight at the deadline is sent
after termination. A step that an abort listener starts at the deadline
runs. A finite remaining time above the setTimeout limit is passed to
setTimeout as is. The three tests fail on this commit.
- processQueue drops queued updates once the invocation terminates.
  checkpoint() already refused new ones, but updates queued behind a
  batch in flight were still sent when that batch returned.
- TerminationManager.terminate sets its flag before it runs the
  checkpoint callback. The callback aborts the invocation signal, so an
  abort listener that starts an operation now sees the termination.
- The deadline timer caps each delay at MAX_TIMER_DELAY_MS. setTimeout
  turns a larger delay into 1 ms, so a long finite deadline made the
  timer fire and re-arm every millisecond.

The no-deadline test now checks that no deadline timer is set, so it
fails if the Number.isFinite guard is removed.
@ParidelPooya
ParidelPooya requested a review from a team as a code owner November 10, 2026 01:10
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 10, 2026 01:11 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The previous commit made processQueue drop queued updates once the
invocation terminates. A fault termination, such as a context
validation error, needs those updates: the service records the failed
execution with them. So three context-validation examples failed.

The drop is removed, with its test. After the deadline, an update that
was queued behind a request in flight can still be sent. On Lambda
Managed Instances the service rejected such a late update with
"Invalid checkpoint token" once the retry ran.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 10, 2026 17:34 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 10, 2026 18:34 — with GitHub Actions Active
@github-actions

This comment has been minimized.

…-after-termination

# Conflicts:
#	packages/aws-durable-execution-sdk-js/src/handlers/run-in-child-context-handler/run-in-child-context-composed.test.ts
#	packages/aws-durable-execution-sdk-js/src/handlers/run-in-child-context-handler/run-in-child-context-serdes.composed.test.ts
#	packages/aws-durable-execution-sdk-js/src/with-durable-execution-queue-completion.test.ts
#	packages/aws-durable-execution-sdk-js/src/with-durable-execution.plugin-lifetime.composed.test.ts
A step that ignores its signal and returns after the deadline still
runs its serdes. A serdes can write the result somewhere, so the old
invocation can overwrite what the retry stored. The test fails on this
commit.
A step that returns after the deadline still ran its serdes before the
refused checkpoint. A serdes can write the value somewhere, such as a
file keyed by the step, so the old invocation could overwrite what the
retry stored. safeSerialize now returns a promise that never settles
once the invocation terminated. Every handler serializes through it.

Test fixtures:
- Hand-written TerminationManager mocks in the handler tests get
  isTerminationRequested. No expectation changed.
- Two ReplayChildren fixtures in the run-in-child-context composed tests
  had no Name or SubType. Replay validation terminated them as
  non-deterministic, and they passed only because work went on after the
  termination. They now record the name and subtype the code uses.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 11, 2026 02:04 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 12, 2026 20:35 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 12, 2026 20:42 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown

Codex AI review

Two prior findings remain: checkpoint draining beyond the deadline (packages/aws-durable-execution-sdk-js/src/with-durable-execution.ts:628, previously deferred) and delayed callbacks starting after termination (packages/aws-durable-execution-sdk-js/src/context/durable-context/durable-context.ts:362). Static review only; tests were inspected but not run.

Suppressed repeats

  • arf_v1_ie3jzjt5n2wks655b3hld4ert6 — deferred (maintainer verdict)

Reviewed commit 529b9fe41838932b08d995d6553c1379b068d802. Workflow run

@github-actions

Copy link
Copy Markdown

Slow integration tests

Slow means ≥1.5x the median + 1s, or ≥5s over the median. The median is from recent runs.

Test file Node.js 22.x Node.js 24.x
microvm/session/microvm-session.test.ts 43.1s (median 38.0s) -

Run details

@ParidelPooya
ParidelPooya merged commit 91ccd4a into main Nov 12, 2026
51 checks passed
@ParidelPooya
ParidelPooya deleted the fix/sdk-no-operations-after-termination branch November 12, 2026 21:37
yaythomas pushed a commit that referenced this pull request Nov 13, 2026
## Issue Link

Closes #1101

## Description

After an invocation returns, unfinished work could still submit
checkpoints because `CheckpointManager.dispose()` did not close
checkpoint admission. Disposal now reuses `setTerminating()` to refuse
new requests, and `processQueue()` drops unsent queued work. Ordinary
termination still drains its queue. Requests already in flight retain
their completion behavior, including force-checkpoint waiters and
accepted execution-result checkpoints.

Invocation cleanup also aborts `StepContext.signal`, allowing steps the
handler did not await to stop. The public signal documentation now
includes normal return and error exits. `checkAndTerminate()` refuses to
arm a termination timer after disposal.

The deadline cleanup test introduced by #1202 now checks two independent
outcomes: the signal is aborted when the invocation returns, and
advancing beyond the original deadline does not call `terminate()` with
`INVOCATION_DEADLINE_PASSED`. Fake `Date` and timers advance together;
`setImmediate` remains real so checkpoint processing can drain. Runtime
behavior is unchanged by this test correction.

## Design considerations

- Reuse the existing admission gate rather than adding disposal checks
to each checkpoint entry point. The dispatch guard remains necessary
because normal termination must drain queued requests while disposal
abandons them. This retains the lifecycle approach previously approved
in this PR.
- Do not clear the queue immediately in `dispose()`: that would discard
force-checkpoint waiters attached to an active request.
- The old deadline test used an unaborted signal as a proxy for timer
cleanup. Completion now intentionally aborts that signal. Merely
changing `false` to `true` would lose timer-cleanup coverage, so the
test observes deadline termination directly. Removing completion abort
would leave unfinished steps without their stop signal.

## Validation

Fresh local checks for the deadline-test and documentation update:

- Reproduced the original failure on `1f3ccda6`: `does not abort the
signal after the invocation returned` failed with `expected true to be
false` (1 failed, 7 filtered out).
- Revised deadline and step-abort composed suites: 14 tests passed.
- Temporarily removing `clearTimeout(deadlineTimer)` makes the revised
test fail on an unexpected `INVOCATION_DEADLINE_PASSED` call.
Independently removing the cleanup `terminated.abort()` makes it fail on
the signal assertion. Both runtime lines were restored before final
validation.
- Complete SDK suite with coverage: 1,604 tests passed across 123 files
on Node 22.23.4 and Node 24.21.0, including the SDK contract tests.
Coverage thresholds passed.
- SDK typecheck, lint (220 files), ESM/CommonJS/declaration builds, and
`git diff --check` passed. The first build attempt lacked a shared
Rollup plugin because only core dependencies were installed; installing
the additional workspace dependencies from the lockfile resolved the
local setup issue.

The existing disposal tests cover refused admission, queued work
abandoned before dispatch, in-flight completion, execution-result
outcomes, termination draining, and absence of termination after
disposal. The earlier admission regression evidence recorded in this PR
showed two failures on pre-fix parent `48e75ca4`, then two passes with
the admission fix.

The existing OTel installed-consumer typecheck case has a 30-second
timeout, matching its adjacent compiler test. In [the failed Node 22
job](https://github.com/aws/aws-durable-execution-sdk-js/actions/runs/43302215706/job/131481296020),
compilation exceeded Vitest's default five-second limit. Its strict
typechecking and assertions are preserved; all seven cases were verified
on Node 22.23.4 in the prior timeout follow-up. That package was not
rerun for this core test/documentation update.

## Scope and unverified behavior

- An already in-flight checkpoint can finish after disposal. The
separate polling-timer continuation in `forceRefreshAndCheckStatus()`
remains outside this fix; subsequent checkpoint dispatch is blocked.
- The existing deadline test's expectation changes because normal
completion now aborts the signal. Its original timer-cleanup regression
coverage is retained. No protected SDK invariant test is changed.
- Cloud integration and conformance were not run locally. Live Lambda
behavior remains unverified by these local checks. The cause of the
earlier installed-consumer compilation slowdown is also unverified.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

---------

Co-authored-by: Pooya Paridel <ParidelPooya@users.noreply.github.com>
Co-authored-by: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 529b9fe4 Deployed Nov 12, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #3070
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants