Skip to content

[Bug]: ExecutionOtelPlugin re-exports completed operations on later replays #941

Closed

Description

@zhongkechen

Expected Behavior

With ExecutionOtelPlugin, an externally completed durable operation should export its completed operation span when completion is first observed. Subsequent invocations that replay the stored outcome should not export that completed span again.

This applies to completed waits, invokes/chained invokes, and callbacks. Deterministic span IDs preserve identity; they should not require the exporter or backend to deduplicate repeated records.

Actual Behavior

ExecutionOtelPlugin.onOperationEnd creates and ends a recording span even when the hook is a replay of an already completed operation. Each later invocation can therefore export another record with the same trace ID and span ID.

The September 15 cloud investigation reproduced this with core SDK 2.4.0 and OTel 1.0.0:

  • One completed invoke produced five exported records with the same span ID, each retaining its 801 ms duration.
  • Across three cloud executions, 18 invocations, and 46 completed wait/invoke/callback observations, every observation retained its end timestamp.

Duplicate export is confirmed. Duration growth was reported separately but was not reproduced by this investigation and is not established by this issue.

Steps to Reproduce

  1. Configure a real Lambda durable function with ExecutionOtelPlugin and capture raw exported spans across invocations. Use a collector or exporter that retains individual records before any backend deduplication.
  2. Run a named wait, invoke, or callback that completes across an invocation boundary.
  3. After that operation has completed, perform additional sequential durable waits long enough to force further suspensions and resumes. Use static operation names.
  4. For the first operation, correlate raw records by durable.execution.arn, durable.operation.id, trace ID, and span ID.
  5. Expected: one completed operation span across the first completion and subsequent replays. Actual: later invocations export the completed operation span again.

Repeat for wait, invoke/chained invoke, and callback operations. A test that stops immediately after the first completion will miss the defect.

SDK Version

Cloud reproduction: @aws/durable-execution-sdk-js 2.4.0, @aws/durable-execution-sdk-js-otel 1.0.0.

Source rechecked on October 1 at main commit 74bacbf7e2ed2009f5653d6ab61d193582aa7093; the missing replay guard is still present. The cloud reproduction was not rerun against this commit.

Node.js Version

The exact runtime version was not captured in the available cloud investigation summary.

Is this a regression?

Unknown; no last working version has been established.

Additional Context

Suggested Fix and Acceptance Criteria

  • Add the appropriate replay guard in ExecutionOtelPlugin, preserving first observations of external completions (isReplay: false).
  • Add focused regression coverage for first completion followed by multiple later resumes, checking raw export counts and stable operation identity and timestamps.
  • Cover successful and failed terminal outcomes where applicable, and ensure genuine new operations and attempts still export correctly.
  • Validate through the local harness as well as the cloud path once the companion input-wrapper metadata bug is fixed. Local metadata loss can otherwise suppress the first completion after a replay guard is added.
  • Coordinate additional language-neutral replay/export coverage in the separate conformance repository.

Activity

  1. zhongkechen commented on Oct 15, 2026

    @zhongkechen
    ContributorAuthor

    Fixed by #953 together with the local-input metadata correction in #954; both are merged.

    For the reported completion-between-invocations followed by later successful replays, main now retains the first external completion, drains received completions that traversal did not reach, and skips later replay-only exports. The wrapper preserves UpdatedOperationIds so the first completion is not lost in local tests. Existing public SDK regressions cover waits, successful/failed invokes and callbacks, multiple resumes, raw export counts, stable identity/supplied timestamps, and unchanged step side effects. The recorded cloud case 26 proof additionally covers a root callback completing between invocations followed by later replays.

    Closing the repeated-export bug described here. This does not promise exactly-once delivery after failed-invocation update redelivery, recovery of metadata pruned by the service, or arbitrary shutdown/flush delivery. Those additional lifecycle/conformance cases remain tracked in #960 and aws/aws-durable-execution-conformance-tests#131; virtual FLAT-map contexts remain separate in #997.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageIssue needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions