Repository navigation
Description
Expected Behavior
With ExecutionOtelPlugin, an externally completed durable operation should export its completed operation span when completion is first observed. Subsequent invocations that replay the stored outcome should not export that completed span again.
This applies to completed waits, invokes/chained invokes, and callbacks. Deterministic span IDs preserve identity; they should not require the exporter or backend to deduplicate repeated records.
Actual Behavior
ExecutionOtelPlugin.onOperationEnd creates and ends a recording span even when the hook is a replay of an already completed operation. Each later invocation can therefore export another record with the same trace ID and span ID.
The September 15 cloud investigation reproduced this with core SDK 2.4.0 and OTel 1.0.0:
- One completed invoke produced five exported records with the same span ID, each retaining its 801 ms duration.
- Across three cloud executions, 18 invocations, and 46 completed wait/invoke/callback observations, every observation retained its end timestamp.
Duplicate export is confirmed. Duration growth was reported separately but was not reproduced by this investigation and is not established by this issue.
Steps to Reproduce
- Configure a real Lambda durable function with
ExecutionOtelPluginand capture raw exported spans across invocations. Use a collector or exporter that retains individual records before any backend deduplication. - Run a named wait, invoke, or callback that completes across an invocation boundary.
- After that operation has completed, perform additional sequential durable waits long enough to force further suspensions and resumes. Use static operation names.
- For the first operation, correlate raw records by
durable.execution.arn,durable.operation.id, trace ID, and span ID. - Expected: one completed operation span across the first completion and subsequent replays. Actual: later invocations export the completed operation span again.
Repeat for wait, invoke/chained invoke, and callback operations. A test that stops immediately after the first completion will miss the defect.
SDK Version
Cloud reproduction: @aws/durable-execution-sdk-js 2.4.0, @aws/durable-execution-sdk-js-otel 1.0.0.
Source rechecked on October 1 at main commit 74bacbf7e2ed2009f5653d6ab61d193582aa7093; the missing replay guard is still present. The cloud reproduction was not rerun against this commit.
Node.js Version
The exact runtime version was not captured in the available cloud investigation summary.
Is this a regression?
Unknown; no last working version has been established.
Additional Context
- Companion local-harness metadata bug: [Bug]: Local invocation input wrapper drops UpdatedOperationIds and misclassifies first completions as replays #942.
DurableExecutionInvocationInputWithClientdropsUpdatedOperationIds, which can misclassify first completions as replays in local tests. ExecutionOtelPlugin.onOperationEndonly checks whether tracing is enabled before creating the completed span. It does not inspectinfo.isReplay.InvocationOtelPlugin.onOperationEndalready skips replayedWAIT,INVOKE,CHAINED_INVOKE, andCALLBACKcompletions.- [otel] End all recording spans on non-terminal invocations #831 / fix(otel): end spans on non-terminal invocations #882 addressed recording-span lifecycle and merged before the cloud reproduction above; they did not prevent these duplicate completed-operation exports.
- Workflow-parent selection is a separate design topic governed by otel: allow propagated or same-trace ambient Workflow parent aws-durable-execution-conformance-tests#99. Fixing duplicate exports does not require changing parent selection.
Suggested Fix and Acceptance Criteria
- Add the appropriate replay guard in
ExecutionOtelPlugin, preserving first observations of external completions (isReplay: false). - Add focused regression coverage for first completion followed by multiple later resumes, checking raw export counts and stable operation identity and timestamps.
- Cover successful and failed terminal outcomes where applicable, and ensure genuine new operations and attempts still export correctly.
- Validate through the local harness as well as the cloud path once the companion input-wrapper metadata bug is fixed. Local metadata loss can otherwise suppress the first completion after a replay guard is added.
- Coordinate additional language-neutral replay/export coverage in the separate conformance repository.
Activity
fix(sdk): preserve updated operation IDs in local invocation inputs (#…
fix(sdk): skip exporting replayed external completions (#953)
zhongkechen commentedon Oct 15, 2026
Fixed by #953 together with the local-input metadata correction in #954; both are merged.
For the reported completion-between-invocations followed by later successful replays, main now retains the first external completion, drains received completions that traversal did not reach, and skips later replay-only exports. The wrapper preserves
UpdatedOperationIdsso the first completion is not lost in local tests. Existing public SDK regressions cover waits, successful/failed invokes and callbacks, multiple resumes, raw export counts, stable identity/supplied timestamps, and unchanged step side effects. The recorded cloud case 26 proof additionally covers a root callback completing between invocations followed by later replays.Closing the repeated-export bug described here. This does not promise exactly-once delivery after failed-invocation update redelivery, recovery of metadata pruned by the service, or arbitrary shutdown/flush delivery. Those additional lifecycle/conformance cases remain tracked in #960 and aws/aws-durable-execution-conformance-tests#131; virtual FLAT-map contexts remain separate in #997.