Skip to content

[Testing]: Add LMI cloud regression tests for concurrency, lifecycle and OpenTelemetry #1017

Open

Description

@zhongkechen

Category

Integration Tests — missing regression coverage for Lambda Managed Instances.

Description

Add dedicated local regression and real-LMI cloud coverage for JavaScript/TypeScript Durable Execution invocation concurrency, deadlines, checkpoint admission, cleanup, OpenTelemetry invocation isolation, and invocation-local X-Ray propagation.

Implementation PR: #928 — shared persistent LMI regression coverage.

SDK defect: #927. This issue tracks the test suite and its evidence. Runtime remediation remains tracked by #927, including the implementation proposed in #1014.

Observability validation ownership: This open testing issue now owns the remaining deployed-LMI acceptance for OTel isolation #950 and X-Ray carrier #951. Their implementation changes landed in #924, including the carrier work from #955, and the defect issues are closed. Their closure does not complete the cloud validation tracked here. Validate compatible JS Core 3 / OTel 2 artifacts with the intended lifecycle implementation from #1014. Coordinate the matching Java acceptance in Java #727 under umbrella Java #798.

Cross-SDK umbrella: aws/aws-durable-execution-sdk-java#798. Equivalent cloud-testing work is tracked in Java #727 and Python #742.

The baseline LMI example coverage described in #928 exercises a basic step but does not establish the lifecycle contract reported in #927. LMI remains active between invocations, and a platform timeout does not forcibly terminate old code. Concurrent invocations can also share a Node.js worker. Tests must distinguish root/parent completion, SDK wrapper return, disposal, PENDING, and the actual platform deadline.

Affected Files or Areas

  • lmi-tests/: deployment, controlled fixtures, local regressions, cloud driver, evidence collection, and harness tests introduced by test: add LMI lifecycle and OTel regression coverage #928.
  • JavaScript example cloud-test routing and the LMI integration workflow.
  • Public withDurableExecution, step/child/map/parallel/promise APIs, checkpoint/disposal boundaries, and suspension/replay behavior.
  • OTel Invocation/Execution plugin factories, invocation-local X-Ray extraction, supported provider/layer configurations, and span/trace evidence collection.

Proposed Improvement

1. Reusable LMI deployment and observable placement

Use the persistent shared-function approach in #928. The proposed initial fixture uses Node.js 24/arm64, one execution environment, native environment concurrency 2, and one Node.js worker. Record the actual deployed runtime, capacity-provider association, concurrency, qualified target, SDK commit, and artifact identity before testing.

Reuse the persistent stack/function and content-addressed artifacts across runs. Serialize deployment, test execution, and settlement; keep per-run controls/evidence isolated. Do not update the shared function while an earlier test execution still uses it. Report unsupported provisioning, unavailable capacity, and inability to establish placement separately from SDK assertion failures.

Parallel driver requests alone are insufficient. Require distinct request/execution IDs with overlapping intervals in the same environment and Node.js worker. Keep native invocation concurrency separate from SDK branch maxConcurrency.

2. Required scenarios

Scenario Required evidence and assertions
Basic execution and existing examples Preserve the selected examples' inputs, real checkpoint behavior, results, and history assertions when routing them through the shared LMI function.
Same-worker invocation overlap Independent executions with overlapping lifetimes retain their own results, tokens, and operation histories; a healthy peer continues making progress.
Child/map/parallel progress Exercise nested durable work and constrained branch concurrency through public SDK APIs. Do not assume Java executor APIs exist in JS.
Early completion Cover promise.race, promise.any, and early-completing map/parallel while another branch is held. Distinguish winner selection from wrapper return; observe late user effects and SDK requests separately.
Root return/failure with work in flight Release held work only after the wrapper boundary under test. Verify the chosen admission/cancellation contract and absence of new SDK checkpoint dispatch after disposal/return.
Deadline during a step or checkpoint Cross the real configured invocation timeout while holding step I/O or delivery of a real checkpoint response. Correlate the original request, later body admission, attempted requests, service history, retry, and capacity recovery.
Disposal and timer races Supplement public cloud cases with controlled local tests for idempotent disposal, queued callbacks, checkpoint/force-checkpoint admission, and in-flight refresh attempts to rearm polling.
Suspension and replay Resume through actual durable waits/callbacks. Completed bodies remain skipped; stored successes/failures and operation identity remain stable. Do not force the original JS business stack's catch/finally to run merely because the invocation returned PENDING.
Warm reuse and cleanup Repeat bounded success/failure/suspend/resume cases and attribute remaining SDK timers, callbacks, requests, or retained resources to their owning invocation. An unreachable pending Promise alone is not evidence of a leak.

The current Core 3 #928 catalog selects 15 example suites (25 tests), including the runtime-carrier example, plus 12 lifecycle cases. Preserve their behavioral assertions as the suite evolves. These counts and the single carrier example do not establish the additional OTel acceptance below.

3. Timing, evidence, and test isolation

  • Separate Lambda invocation timeout, overall durable execution timeout, and test-driver/collection deadlines. Local fake-clock tests do not establish deployed timeout behavior.
  • Use gates/barriers and request identities to prove the intended race. An old release, retry attempt, spare peer slot, or replacement environment must not accidentally satisfy the original invocation's recovery assertion.
  • Record attempted checkpoint/transport requests separately from service-accepted operations. A delayed response fixture must preserve the real service response and must not fabricate remote acceptance.
  • Retain deployment/readback manifests, JUnit, lifecycle ledgers, request/worker/environment identities, logs, and execution histories, including on failed runs.
  • Release controls only after the relevant assertions, then settle or stop unfinished executions and require quiescence before reusing the fixture. Failed settlement must prevent subsequent cases from using a contaminated shared function.
  • Bound fixture escapes, provisioning, placement, collection, and cleanup. Test-only release or emergency escape cannot count as successful recovery. The proposed five-second cleanup/recovery target is a test criterion to agree with [Bug]: LMI invocations can outlive deadlines and continue checkpointing after disposal #927, not a product SLA or a promise to forcibly stop arbitrary I/O.

4. CI and regression reporting

Run unprivileged harness validation independently of AWS-backed tests. Preserve existing restrictions for forked PRs and Dependabot, and document the workflow triggers and manual dispatch procedure. Serialize access to the persistent cloud fixture.

Keep desired-behavior regressions as ordinary assertions. An unfixed SDK baseline may fail those assertions; report this distinctly from fixture, deployment, placement, or collection errors. Validate the same assertions with the intended SDK fix and retain the failing baseline evidence. Do not make failures disappear through weakened checks, recovery through unrelated capacity, or retries until a test passes.

5. Deployed LMI OpenTelemetry and X-Ray acceptance

The observability scenarios are additional acceptance work. Exercise the real public SDK wrapper, plugin factory registration, both Invocation and Execution OTel views, and a real provider/exporter. Retain the plugin-free lifecycle controls so failures can be attributed to the SDK lifecycle, plugin, runtime carrier, or export/collection path. Passing the core lifecycle suite, merging an implementation PR, or passing ordinary on-demand OTel conformance does not establish this LMI acceptance.

  • Invocation isolation under proven overlap. Use the same durable operation names/sequences in independent executions, with distinct execution ARNs and upstream trace identities. Establish overlap with bounded barriers and correlate runtime request IDs, execution IDs, environment IDs, and the runtime sharing boundary described below. Assert that operation/span attribution, parent relationships, invocation duration/status and per-invocation buffers belong to the correct execution, without unexpected duplicate or cross-attributed Invocation/Workflow spans.
  • Lifecycle, replay and late activity. Exercise success, handler failure, real suspension/resume, retry and actual invocation timeout while a peer remains active. Hold a completion across another invocation's End/cleanup boundary through the public execution path. Ending or resetting invocation A must not clear, end, relabel or otherwise corrupt B's spans/context. Retain checkpointed successes/failures and expected side-effect counts on replay; do not force a suspended JS business stack to unwind.
  • Shared provider/exporter ownership. Reuse the environment-owned provider/exporter across invocation-local plugin instances. Check that one invocation cannot shut down the shared resources or block a healthy peer indefinitely. Include bounded exporter failure/stall and flush-budget cases, observing the documented SDK contract. Report expected timeout/export losses explicitly; do not introduce an unconditional lossless-delivery or export-acknowledgment guarantee.
  • Invocation-local X-Ray propagation. On deployed LMI, record the genuine runtime-provided header at invocation entry and verify the documented trace ID (Root), parent (Parent) and explicit sampling (Sampled=0/1) behavior in both views, including overlapping executions and real resume/retry. Distinct headers must not bleed between invocations or inherit stale process-wide carriers. Check propagation at concurrency 1 as well as under proven overlap. Keep sampling expectations consistent with the selected view and current invocation's runtime header.
  • Carrier and compatibility matrix. Retain controlled public-wrapper/installed-artifact coverage for unavailable, absent, empty, malformed and unreadable carriers, ordinary-Lambda fallback, supported custom extractors and older runtime/context interfaces. An available invalid/empty/unreadable carrier must not borrow a stale global header. If a carrier condition cannot be produced by the real LMI runtime, label that case as a controlled compatibility fixture; it is not evidence of a native runtime case. Required genuine-runtime cases remain separate acceptance gates.
  • Evidence and regression sensitivity. Record exact core/plugin/provider/layer versions and checksums, tested commits, function/version/configuration readbacks, runtime and environment identities, correlated request/execution/trace/span IDs, exported spans or trace-query results, lifecycle ledgers, JUnit and durable histories. Use bounded collection and distinguish placement/setup, SDK/plugin assertions, sampling, exporter failure and missing evidence. Retain comparable negative controls against a pinned affected implementation or deterministic defect fixture; a missing span or failed placement cannot count as an isolation pass.

Reuse the existing persistent LMI fixtures and evidence collection where practical, adding dedicated observability handlers/assertions as needed. Preserve the existing replay, checkpoint, cleanup and concurrency assertions. Keep this testing issue open until both its original lifecycle requirements and the added observability acceptance below are satisfied; required unverified cases remain unchecked.

Node.js sharing and carrier boundary: Prove simultaneous invocations in the same execution environment and Node.js worker, using worker identity and overlapping request intervals. A provider/exporter shared only across separate workers does not exercise the reported race. Capture context.xRayTraceId through the runtime entry point and verify the immutable invocation snapshot reaches the registered plugin/extractor without shared environment-variable writes. Retain supported runtime-worker configuration and ordinary-Lambda compatibility controls; distinguish a genuine runtime header from an injected fixture.

Acceptance Criteria

  • PR test: add LMI lifecycle and OTel regression coverage #928 is linked to this testing issue; [Bug]: LMI invocations can outlive deadlines and continue checkpointing after disposal #927 remains the runtime-defect tracker.
  • The shared LMI deployment verifies the exact SDK artifact and actual native runtime/concurrency configuration.
  • Same-worker overlap and affected-capacity recovery are established by invocation/environment identities, not only driver-side concurrency.
  • The scenarios above exercise real public SDK behavior, including initial invocation and replay/resume, with stored success/failure and side-effect assertions.
  • Controlled local regressions reproduce the relevant [Bug]: LMI invocations can outlive deadlines and continue checkpointing after disposal #927 boundaries, and the intended SDK fix passes the unchanged applicable assertions.
  • Harness tests cover gate ownership, timing, deployment/readback validation, evidence interpretation, and cleanup/isolation failure paths.
  • A real-LMI run of the intended SDK/test combination retains complete evidence and reports infrastructure/collection problems separately from SDK failures.
  • Commands, prerequisites, workflow triggers, persistent-resource ownership, timing targets, known limitations, and interpretation of negative controls are documented.
  • Both OTel views pass deployed-LMI attribution/isolation assertions with proven same-environment, same-worker overlap and reused operation names/IDs.
  • Success, failure, suspension/resume, retry, actual timeout and controlled late activity preserve the healthy peer's spans/context and the SDK's checkpoint/replay semantics.
  • Shared provider/exporter ownership, bounded flush/failure behavior and expected timeout/export limitations are verified through the public registration path.
  • Genuine LMI runtime headers preserve the required Root/Parent/Sampled behavior at concurrency 1 and under overlap, including sampled/unsampled and resume/retry cases, without cross-invocation fallback.
  • The carrier/compatibility matrix records which cases use genuine runtime inputs and which use controlled fixtures, with compatible installed Core 3 / OTel 2 and provider/layer artifacts.
  • Passing observability run URLs and retained correlated span/trace evidence are linked here with exact deployed artifact/configuration identities and regression-sensitivity evidence; local/core-only results are not substituted for the new LMI gates.

Additional Context

  • PR #928 is the existing implementation proposal for this testing work.
  • #904 addressed Capacity Provider CI deployment/setup stability; it does not cover this lifecycle regression suite.
  • #950 and #951 retain the original defect reports; their remaining deployed-LMI validation is explicitly in this issue's scope. Plugin-free core lifecycle tests do not establish OTel acceptance.
  • Node.js runtime on LMI
  • LMI execution-environment lifecycle

Activity

  1. changed the title [-][Testing]: Add LMI cloud regression tests for concurrency and invocation lifecycle[/-] [+][Testing]: Add LMI cloud regression tests for concurrency, lifecycle and OpenTelemetry[/+] on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pkg:testingPackage: aws-durable-execution-sdk-js-testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions