Skip to content

[Bug]: OTel and telemetry exporter report StepError instead of checkpointed operation error types #1124

Open

Description

@zhongkechen

Expected Behavior

The SDK's shared operation-error metadata should preserve the checkpointed ErrorType for both instrumentation plugins:

  • OpenTelemetry (@aws/durable-execution-sdk-js-otel): both execution and invocation views should record it as exception.type, as required by G01.
  • Durable Execution Telemetry (@aws/durable-execution-sdk-js-telemetry-exporter): operation records should retain it in operations[].error.name and, when present, operationsByName.<name>.error.name. Every configured exporter destination receives these records.

A user or external error should not be relabeled as StepError merely because the SDK reconstructs it for instrumentation. The defect is in shared SDK metadata construction, so this issue covers both plugins.

For the reported scenarios, both the execution and invocation views should record:

Scenario Span Expected exception type
S12: child-context function throws new Error(...) directly otel-failed-child-context Error
S17: callback is failed externally with ErrorType: "RejectedError" otel-failed-callback-callback RejectedError
S18: invoked durable handler throws new Error(...) directly otel-failed-invoke Error

The parent otel-failed-callback span should continue to record CallbackExternalError. Caller-facing SDK errors, including CallbackExternalError, ChildContextError, and InvokeError, should retain their existing behavior.

Actual Behavior

The reported X-Ray traces show StepError for all three spans above even though the stored failures contain ErrorType: "Error" or ErrorType: "RejectedError". Custom types that are not defined by the SDK encounter the same fallback.

There is also a delivery-path inconsistency for callbacks: the live completion hook replaces the stored diagnostic error with CallbackExternalError, while stored/replayed metadata is reconstructed as StepError. Neither represents the external RejectedError on the callback leaf.

The OTel plugins pass these errors to span.recordException(...), so their names become exported exception.type values.

The telemetry exporter also consumes the same checkpoint-derived OperationInfo.error. It writes the error's own name and cause without unwrapping them. As a result, an operation whose stored type is RejectedError is exported with error.name: "StepError" and error.cause.name: "RejectedError". The original type is present in the synthetic cause, but the top-level operation error is mislabeled. This affects all telemetry exporter destinations and both operation record formats.

Steps to Reproduce

Run each scenario separately with the execution OTel plugin and the invocation OTel plugin, using an in-memory span exporter or an OTel backend such as X-Ray.

  1. Call runInChildContext("otel-failed-child-context", async () => { throw new Error("child failed"); }). Throw directly from the child function, rather than from a step inside it.
  2. Call waitForCallback("otel-failed-callback", submitter). Fail its callback with ErrorType: "RejectedError" and an error message. Exercise both completion while the invocation is active and completion after suspension.
  3. Call invoke("otel-failed-invoke", qualifiedTarget, input), where the target durable handler directly throws new Error("target failed").
  4. Inspect each operation's stored ErrorType and the corresponding span's exception event. The stored type and exception.type differ.
  5. Catch the failures and suspend at a later durable operation, then resume. Verify that completed work is skipped and the original diagnostic type is preserved in replay metadata.

The existing OTel conformance scenarios S12, S17, and S18 exercise the reported cases.

For the telemetry plugin, repeat these scenarios with durableExecutionTelemetry and a capturing exporter. Compare the stored ErrorType with operations[].error.name and operationsByName.<name>.error.name where available, including the cause shape on initial execution and replay. This SDK-to-exporter path also needs regression coverage.

SDK Version

Source revision: 6d024cbe7ee6a95e8d814eb8a58d5c5bb41d67ab

  • @aws/durable-execution-sdk-js: 3.0.0
  • @aws/durable-execution-sdk-js-otel: 2.0.0
  • @aws/durable-execution-sdk-js-telemetry-exporter: 1.0.0

Node.js Version

22.x — local reproduction used Node.js 22.23.4. The runtime version of the originally reported cloud traces is unspecified.

Is this a regression?

Unknown; no last working version has been established.

Additional Context

The error is relabeled during SDK metadata construction:

The telemetry exporter passes each operation error through toErrorRecord, which preserves the supplied name and cause. This confirms that the defect is visible outside OTel.

A fix should reconstruct diagnostic errors from the stored type, message, stack, and error data, while retaining the current caller-facing reconstruction and errorMapper behavior. Missing error details should remain absent.

Compatibility and validation for both plugins

Changing the shared conversion is an observable metadata change for both plugins. For example, a checkpointed external rejection would change from:

{"name":"StepError","message":"rejected","cause":{"name":"RejectedError","message":"rejected"}}

to:

{"name":"RejectedError","message":"rejected"}
  • Telemetry queries and consumers that filter on StepError or read the original type from an operation's error.cause.name may need updating. The local candidate removes the synthetic wrapper/cause; it does not rewrite previously exported records.
  • Custom plugins still receive an Error, but wrapper-specific assumptions such as instanceof StepError, .errorType, or a synthetic .cause can change.
  • Caller-facing exceptions, retry behavior, and InvocationEndInfo.executionError should remain unchanged.
  • The telemetry exporter's documentation of operation causes and invoke error names describe the existing wrapper shape and need updating with the fix.
  • Regression coverage must exercise the real SDK-to-telemetry-exporter conversion as well as both OTel views. Existing exporter unit tests that inject errors directly do not establish compatibility for this change.

Local regression tests reproduced the bug. A local candidate fix passes 1,586 SDK tests and 38 OTel integration cases covering both views, live/resumed delivery, replay, and custom error names. The candidate is not published. SDK-to-telemetry-exporter regression coverage and the documentation updates above remain outstanding, and cloud conformance traces have not been rerun.

Activity

  1. added
    pkg:pluginsPlugin interface, possibly impacting all plugins
    and removed on Oct 29, 2026
  2. changed the title [-][Bug]: OTel operation spans report StepError instead of the checkpointed exception type[/-] [+][Bug]: OTel and telemetry exporter report StepError instead of checkpointed operation error types[/+] on Oct 29, 2026
  3. hln33 commented on Oct 30, 2026

    @hln33
    Contributor

    Severity - medium

    What is wrong. exception.type on OTel spans and operations[].error.name in telemetry exporter records carry StepError instead of the stored type. Any trace filter, alert, or dashboard keyed on exception type reads the wrong value.

    What is correct. Span status, error message, stack trace, execution outcome, retries, and the exception the caller receives are all unchanged. The telemetry exporter keeps the original type under error.cause.name.

    Scope. Every user of @aws/durable-execution-sdk-js-otel or @aws/durable-execution-sdk-js-telemetry-exporter whose failing operation has a non-SDK error type. That is the common case for user code.

    Not a regression. The fallback predates the first release of both plugins. The plugins have always emitted this value.

    Why not higher severity There is no ongoing outage, no data loss, and no change to execution behavior. A customer can still identify the failure from the message.

    Why not lower severity. The defect affects every plugin user on a standard OTel attribute. The fix changes the exported metadata shape, so each release that ships the current shape raises the cost of changing it later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpkg:pluginsPlugin interface, possibly impacting all plugins

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions