[zhongkechen]

The shared LocalDurableTestRunner fix landed in #1020. This PR targets main and includes the Core 3 / OTel 2 factory migration from #924.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Related to #927. LMI acceptance coverage is in #928. This PR does not close #927.

Description

On Lambda Managed Instances, an invocation can continue after its timeout or after the handler returns. The SDK currently admits later durable work and can send checkpoints after disposal. This change gives each invocation its own deadline and admission state. Expiry interrupts it without checkpointing a business failure; normal completion closes admission while already queued checkpoints drain.

The deadline covers plugin loading and factory creation, execution-state pagination, handler work, checkpoint draining, and instrumentation waits. Native monotonic time and captured timers keep the local runner's virtual durable clock from consuming the invocation budget. Operation entry, deferred step/child/condition callbacks, plugin dispatch, awaited continuations, and durable result settlement share the owning invocation's fence. Disposal clears scheduled work; late responses cannot restart polling, publish state, or resume the closed invocation's user stack. Eager operation startup remains intact. When deadline disposal releases a checkpoint-drain waiter, both the normal-result path and the caught-error path recheck the deadline before converting the outcome. Result getters, toJSON and custom error accessors cannot start after expiry; existing withdrawn-token/PENDING and unrecoverable-error decisions keep their priority.

The SDK creates each plugin from the same invocation snapshot passed to Start, including the safe invocation-local X-Ray carrier. Factory creation and the resulting runner use the invocation's deadline. Built-in OTel instances remain separate across invocations while their provider environment is shared. An expired invocation dispatches End once to admitted plugins without waiting past the deadline; overlapping invocation B keeps its own spans live when A ends. The main branch's OTel implementation is unchanged.

Unfinished work reports RETRYING. A fully prepared terminal success or failure selected before expiry survives a later instrumentation-cleanup timeout, and remaining End hooks receive the same status, result and original error. Selection occurs only after serialization, final checkpoint/termination checks and a live deadline gate. Service-requested PENDING and acknowledged oversized results retain their existing precedence. Once an oversized result is acknowledged, deadline End includes the original handler result even if an operation-state hook is still pending; unacknowledged output is never exposed. Built-in OTel spans close before flushing.

DurableExecutionClient gains an optional third request-options argument carrying an invocation-owned AbortSignal. The Lambda transport forwards it to individual send calls and checks it after lazy loading; it does not destroy a shared client. Existing one- and two-argument transports remain compatible. Handler/operation APIs, checkpoint formats and replay identities are preserved. Suspension keeps the original business stack suspended; replay preserves stored successes and failures.

The SDK cannot preempt synchronous JavaScript, cancel arbitrary application I/O, undo an accepted request, or force a custom transport to honor the signal. Already-started side effects still require idempotency. Deadline export remains best effort.

Testing

Validation uses main 721217ae and the current reviewed candidate.

• Full workspace builds and CI lint passed. Full Core: 1,539 passed; Testing SDK: 1,444 passed with 3 existing skips; OTel: 799 passed; Extras: 218 passed.
• The Core declaration build type-checks the new composed tests as well as production source.
• Independent review confirms the guard preserves the existing checkpoint, ACK, PENDING and deadline outcome priorities.

The new public-wrapper regressions hold the terminal checkpoint of an awaited child context. Caught-error controls require no custom message or name accessor after expiry, original-error identity in live FAILED End metadata, and the same FAILED wire error after a normal drain and replay. Both plugin and no-plugin modes reject on deadline without invoking the returned object's toJSON, including after the late checkpoint responds. Positive controls drain normally, serialize once per invocation, and replay the saved child without rerunning its body. Existing acknowledged-output, service-requested PENDING, checkpoint failure and disposal controls retain their outcomes.

The prior Core 2 composition of #1014 and #928 passed all 14 local lifecycle assertions and 12 deployed lifecycle cases in the closed validation PR #1044, with all 184 ledger records settled and no quarantine. That result is historical: fresh composed LMI acceptance is being prepared for this Core 3 integration. No new cloud result is claimed by these local checks.

Demo/Screenshots

Not applicable; public-wrapper, composed lifecycle and original example regressions exercise the behavior.

Checklist

• PR template completed
• Local tests, builds, types and CI lint gates pass
• Current-head GitHub checks and required independent approval
• Fresh composed Core 3 LMI lifecycle acceptance completed

The latest upstream sync includes main 721217ae, including the Extras MicroVM resume-before-delivery checkpoint fix. This review follow-up changes the Core runtime, so the composed LMI bundle and its local contracts must be rebuilt before fresh cloud acceptance. Earlier payload identity is not reused for this candidate.

[zhongkechen]

Merged current main db6f8436 and retained Draft status at 0591adbe. The runtime implementation is unchanged by this review follow-up. Three incoming #960 public telemetry fixtures now exercise the lifecycle fence rather than waiting forever for a detached durable stack: queued completions still export, admitted attempts close with the same identity, and new work attempted after handler settlement reaches neither its user callback nor the real checkpoint transport. All standalone hook-level flush/ancestry tests remain intact.

Final validation: 1,387 Core tests, 1,348 Testing tests (three existing skips), and all 640 OTel tests pass, along with builds, full Core/OTel types and repository lint. The corrected #928 candidate's 14 local lifecycle/manager assertions also pass unchanged when composed with this implementation. One unchanged external-completion case observed an extra resume during the first full rerun; its unchanged 26-test suite and a subsequent complete 640-test run both pass. No assertion was loosened for that observation.

Deployed LMI deadline/recovery/overlap acceptance remains outstanding. This does not cancel arbitrary application I/O, interrupt synchronous JavaScript, or guarantee reclamation from custom transports that ignore the invocation signal. The broader #927 requirements remain open; no merge/readiness/release action was taken.

[zhongkechen]

The recurring extra-invocation failure was traced to a real LocalDurableTestRunner race: after receiving a terminal handler response, invocation tracking became inactive before asynchronous history recording finished. A queued callback wakeup could start another invocation in that gap. The Testing SDK now closes invocation admission before terminal response/error recording or terminal execution-state publication, and ignores queued wakeups after settlement. It still collects final notifications; normal PENDING scheduling, step retries, and existing thrown-error rejection behavior are preserved.

Three deterministic regressions fail against the prior implementation for successful, failed, and throwing handlers, and pass with this fix. Full Testing SDK suites pass across the four affected PR variants (1,378 on the major factory branch; 1,351 on the minor/lifecycle/test-harness branches; three existing skips each), with types and lint passing. The original public OTel one-invocation, side-effect, metadata, and replay assertions are unchanged: all 30 focused cases pass on the rebuilt variants, and the complete carrier OTel suite passes 674 tests.

This change affects local testing orchestration. Production Core lifecycle/compatibility scope, Draft states, merge holds, and deployed acceptance requirements remain unchanged. Fresh CI is running on the published follow-up heads.

[github-actions]

Codex AI review · Finding arf_v1_bt4sq7prdfufxispwg6q3hyul7

[P2] Fence condition-state serialization continuations

This fence ends when check settles. If the subsequent serdes.serialize remains pending past the deadline, releasing it still calls serdes.deserialize and then config.waitStrategy in the disposed invocation. The outer DurablePromise fence suppresses eventual settlement but does not prevent these callbacks; external-storage deserializers can therefore start fresh I/O after timeout and overlap a retry. Fence the serdes continuations and strategy dispatch with the owning lifecycle, leaving them suspended after closure. Add a regression that holds the post-check serializer across expiry and verifies that neither callback starts.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review

One P2 finding at packages/aws-durable-execution-sdk-js/src/handlers/wait-for-condition-handler/wait-for-condition-handler.ts:265: serialization continuations can start work after expiry. Tests were inspected but not executed.

Reviewed commit cd66e4131f52f99aea9ec865f67969badcb30978. Workflow run

[ParidelPooya]

Review of cd66e413

Verdict: needs changes.

The checkpoint fence works. I ran probes on main and on this head. On main, a step that settles after the deadline still runs retryStrategy or serialize, and the SDK sends its checkpoint. On this head, no callback runs and no checkpoint is sent. The full Core, OTel, and Testing SDK suites pass locally.

Blocking

1. Core 3 LMI acceptance has not run. The only cloud evidence is test: validate current LMI lifecycle fix with shared acceptance suite #1044, which tested the Core 2 composition.

2. The waitForCondition finding at wait-for-condition-handler.ts:268 still reproduces. I held serialize across the deadline and then released it. deserialize ran once and waitStrategy ran once after expiry. No checkpoint was sent. Fence lines 281–300, or reply with a disposition.

3. getRemainingTimeInMillis is now called at entry with no guard (with-durable-execution.ts:792, invocation-lifecycle.ts:56). A handler that returns "ok" succeeds on main in each case below. On this head:

   • When the method throws, the invocation rejects with that error.
   • When the member is a number, it rejects with TypeError.
   • When it returns 0, it rejects with InvocationDeadlineError.

   extras (microvm/shared.ts:570-600) treats the first two contexts as valid. This PR changed its own () => 0 test stubs to () => 1000. Real Lambda contexts are unaffected. Suggested fix: read the value the way extras remainingTime() does, and decide what 0 at entry means.

Non-blocking

• Detached work. An operation that is still running when the handler returns is no longer checkpointed, and its span is not exported. This fixes the main-branch bug. It also reverses behavior that four OTel tests asserted, for example a parent span count changing from 1 to 0. Please add a release note.
• Untested fences. I removed each fence and ran the full core suite. No test fails without the step fence (step-handler.ts:337), the waitForCondition fence (:266), the child-context fence (run-in-child-context-handler.ts:378), or the rejection branch of runInvocationCallback (invocation-lifecycle.ts:189). Closing admission on handler resolve is caught only by OTel tests. The two step probes above would serve as regression tests.
• No deadline margin. With a 300 ms budget, the wrapper settled 9.6 ms after the remaining time reached 0. On Managed Instances, Lambda marks a timed-out invocation failed. So the PENDING, acknowledged-result, and prepared-response returns at with-durable-execution.ts:850-861 never reach Lambda. Either reserve a margin, or state that these returns only set the status plugins receive.
• Owner-check comment. The comment at plugin-runner.ts:18-20 says the owner check covers all hook dispatch. invocationOwners is read only in endInvocationOnDeadline.
• Contract. Expiry fences are added one await at a time, and waitForCondition is the eighth gap of this kind. Consider documenting two levels of guarantee. Guaranteed: no backend request and no plugin dispatch after disposal. Best effort: SDK continuations stay pending.

Not run: cloud or LMI tests.

[ParidelPooya]

Thanks for the detailed work on this. The problem in #927 is real. I am asking to close this PR and replace it with three smaller changes. The reasons follow.

Why not continue with this PR

1. The PR went through about 21 review rounds. Three times, a round found no issues and a later round found a new blocking one. The last three rounds each found a new deadline leak in a different place. Under rule 8, this pattern means the design needs to change, not get more fixes.
2. The PR changes a public API. It adds a third argument to DurableExecutionClient and exports DurableExecutionRequestOptions. It also changes how plugin hooks run on the deadline. Rule 3 requires an agreed design note before such code.
3. The PR predates feat(sdk): stop waiting for operation plugin hooks #1067. Since feat(sdk): stop waiting for operation plugin hooks #1067, the SDK no longer waits for operation hooks, and it stops waiting for plugins before the deadline. Much of the plugin-runner fencing in this PR targets the old behavior, and the PR now conflicts with main.
4. The PR adds deadline checks to operation entry, each handler, result getters, toJSON, error accessors and plugin dispatch. Rule 2 asks for the simpler fix at the cause first.

What replaces it

Most of #927 does not need SDK code. Function authors, plugin writers and transport writers can avoid it by following written rules. Two parts are SDK bugs, and no guideline covers them. Each one is a small change at its cause.

1. [Docs]: Guidelines for running durable functions, plugins and transports on Lambda Managed Instances #1103: guidelines for running on Lambda Managed Instances. It covers rules for function authors (step size, I/O timeouts, idempotency, early-completion branches, module state). It also covers the plugin contract (one instance per invocation, late hooks ignored, wrap* calls fn once and inline, bounded exports) and request timeouts for custom transports.
2. [Bug]: CheckpointManager sends checkpoint requests after dispose() #1101: CheckpointManager sends checkpoint requests after dispose(). The fix is one guard in checkpoint(), forceCheckpoint() and processQueue(), with a test that fails first.
3. [Bug]: SDK starts new durable operations after the invocation deadline #1102: the SDK starts new operations after the deadline. The fix is one check where an operation starts. It needs a short design note first, because it decides the status reported on expiry.

These three do not cover branches left running after an early map, parallel or race completion, or transport cancellation through AbortSignal. Each of those needs its own design note if we still want it after the three above land.

@zhongkechen, could you close this PR? Pick up #1101 or #1102 if you want them. The tests you wrote here for those two cases are a good starting point.