Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Demonstrates OTel-enriched logging in a durable execution.

InvocationOtelPluginFactory is the plugin factory the SDK registers; it builds
one InvocationOtelPlugin per invocation. Each plugin installs a logging filter
create_otel_plugin_factory(mode="invocation") selects the factory the SDK registers; it
builds one InvocationOtelPlugin per invocation. Each plugin installs a logging filter
on the root logger (enrich_logger=True by default) when it is constructed. The
filter stamps the active OpenTelemetry trace context (traceId, spanId,
otelTraceSampled) onto every log record that flows through the root handler.
Expand All @@ -17,7 +17,7 @@

from typing import Any

from aws_durable_execution_sdk_python_otel import InvocationOtelPluginFactory
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory

from aws_durable_execution_sdk_python import StepContext
from aws_durable_execution_sdk_python.context import (
Expand Down Expand Up @@ -45,7 +45,7 @@ def greet_in_child(child_context: DurableContext, name: str) -> str:
return result


@durable_execution(plugins=[InvocationOtelPluginFactory()])
@durable_execution(plugins=[create_otel_plugin_factory(mode="invocation")])
def handler(_event: Any, context: DurableContext) -> str:
# Logged at the top level: enriched with the invocation span_id.
context.logger.info("Workflow started")
Expand Down
90 changes: 79 additions & 11 deletions packages/aws-durable-execution-sdk-python-otel/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,28 +42,69 @@ processors, and exporter.

1. Add the [ADOT Lambda Layer](#1-adot-lambda-layer) to your function and set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`
2. Enable [X-Ray Active Tracing](#2-aws-x-ray-active-tracing) on the function
3. Pass `InvocationOtelPluginFactory()` to your handler's `plugins` list
3. Pass `create_otel_plugin_factory(mode="invocation")` to your handler's `plugins` list
4. Add X-Ray write permissions

The SDK's `plugins` list takes plugin *factories*, not plugin instances: it calls
each factory's `create_plugin` once per invocation and the plugin it returns
serves that one invocation. `InvocationOtelPluginFactory` and
`ExecutionOtelPluginFactory` are the factories for the two bundled plugins; each
takes the optional `OtelPluginConfig` that every plugin it builds will use.
serves that one invocation. Import `create_otel_plugin_factory` to choose a view:

```python
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory

execution = create_otel_plugin_factory() # Default: operations parented to Workflow
invocation = create_otel_plugin_factory(mode="invocation") # Operations parented to Invocation
```

Register one of these factories in `plugins`. Explicit `mode="execution"` also
selects the execution view. Pass existing configuration with
`create_otel_plugin_factory(config, mode="invocation")`. Unsupported modes raise
`ValueError` when the factory is selected.

The steps above use `mode="invocation"`, which is the view this guide's trace
examples and screenshots show. `create_otel_plugin_factory()` with no mode
selects the execution view instead, which parents operations to `Workflow` — a
different span tree, not a different amount of instrumentation. See
[Trace Structure](#trace-structure) for both shapes.

Each call constructs a new `ExecutionOtelPluginFactory` or
`InvocationOtelPluginFactory` of the selected view, carrying that class's
`exclusive_group`. Calls are not memoized, so register exactly one: two
factories of the same view in `plugins` are rejected at registration as a
duplicate in the `aws-durable-execution-otel-view` exclusive group. Both existing factory
classes remain supported, with unchanged configuration and behavior.
Comment on lines +70 to +75

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One failure mode isn't documented or tested: plugins=[create_otel_plugin_factory()] combined with DURABLE_EXECUTION_PLUGINS=otel raises PluginLoadError ("registered more than once in exclusive group"). The identity skip in plugin_discovery.py:504 doesn't apply because each call returns a new instance rather than the EXECUTION_OTEL_PLUGIN_FACTORY singleton, so the same-type check at :367 rejects it. The loader's docstring presents "same object passed explicitly and named in the env" as supported, which the selector can't satisfy. Suggest a README sentence (point mixed users at EXECUTION_OTEL_PLUGIN_FACTORY) plus an e2e test.

🤖 AI-assisted comment


Alternatively, install this package in the function artifact or a Lambda layer
and select either OTel plugin by entry-point name:

```text
DURABLE_EXECUTION_PLUGINS=otel
DURABLE_EXECUTION_PLUGINS=otel-invocation
DURABLE_EXECUTION_PLUGINS=otel-execution
```

`otel-invocation` names a default-configured `InvocationOtelPluginFactory`;
`otel-execution` names a default-configured `ExecutionOtelPluginFactory`. The SDK
discovers the selected package entry point at cold start and calls its
`create_plugin` once per invocation, so the handler does not need to import or
explicitly register the plugin.
`otel` and `otel-execution` name the same default-configured
`ExecutionOtelPluginFactory`. The SDK discovers the selected package entry point
at cold start and calls its `create_plugin` once per invocation, so the handler
does not need to import or explicitly register the plugin. This still requires
the plugin package and a configured OpenTelemetry provider and exporter, usually
supplied by the ADOT layer. Installing or importing the package alone does not
enable instrumentation.

**Set exactly one OTel entry name.** Because `otel` and `otel-execution` resolve
to the same factory object, listing both loads one factory: the loader skips the
second by identity and logs

```text
Skipping dynamically configured plugin 'otel-execution' because the same plugin factory is already registered.
```

Listing `otel` with `otel-invocation` names two different views and raises
`PluginLoadError`. The Java and JS/TypeScript SDKs enforce the same one-view
contract but report a redundant second alias in their own idiom — Java rejects
`otel,otel-execution` outright — so do not rely on a multi-alias value behaving
identically across languages.
Comment on lines +103 to +107

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can drop the same sentence here as well: "The Java and JS/TypeScript SDKs enforce the same one-view
contract but report a redundant second alias in their own idiom — Java rejects
otel,otel-execution outright — so do not rely on a multi-alias value behaving
identically across languages."

🤖 AI-assisted comment


OTel 2.0 requires core SDK 3.x and the per-invocation factory contract. Publish
the coordinated core release before the plugin. Core 2.x plugin instances and
Expand Down Expand Up @@ -221,10 +262,10 @@ rejected clearly by core 3.
```python
from aws_durable_execution_sdk_python import DurableContext
from aws_durable_execution_sdk_python.execution import durable_execution
from aws_durable_execution_sdk_python_otel import InvocationOtelPluginFactory
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory


@durable_execution(plugins=[InvocationOtelPluginFactory()])
@durable_execution(plugins=[create_otel_plugin_factory(mode="invocation")])
def handler(event: dict, context: DurableContext) -> dict:
result = context.step(lambda _: fetch_data(event["id"]), name="fetch-data")

Expand All @@ -238,6 +279,9 @@ def handler(event: dict, context: DurableContext) -> dict:
The ADOT layer supplies the global `TracerProvider`; the plugin handles
deterministic ID generation and span lifecycle.

Drop `mode="invocation"` to parent operations to `Workflow` instead of to the
current `Invocation`; see [Trace Structure](#trace-structure).

### 4. Grant Permissions

The function's execution role needs the `AWSXRayDaemonWriteAccess` managed policy (or equivalent permissions) if using X-Ray as the tracing backend.
Expand Down Expand Up @@ -546,6 +590,30 @@ After deploying your function with the plugin configured:

## API Reference

### `create_otel_plugin_factory`

```python
create_otel_plugin_factory(config=None, *, mode="execution")
```

Constructs a new `ExecutionOtelPluginFactory` by default, or a new
`InvocationOtelPluginFactory` for `mode="invocation"`. The optional `config` is
an existing `OtelPluginConfig`, passed unchanged to the selected class. For
example, an application-owned provider can be supplied with
`create_otel_plugin_factory(OtelPluginConfig(tracer_provider=provider))`.

The return value belongs in the handler's `plugins` list. Each invocation gets
its own plugin instance. Selecting a factory does not resolve a provider or
construct an invocation plugin. Mode is selected once; invalid values,
including `None`, raise `ValueError`.

Calls are not memoized —
`create_otel_plugin_factory() is create_otel_plugin_factory()` is `False` — so
register exactly one. Two factories of the same view in `plugins` raise
`PluginLoadError` as a duplicate in the `aws-durable-execution-otel-view` exclusive group. A
`mode` read from configuration at runtime is accepted and type-checks as the
union of both return types.
Comment on lines +610 to +615

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "Calls are not memoized, register exactly one" appears here, in the setup section (:72), and in the function docstring. The new README section is ~70 lines for a small selector. Maybe in a future PR, we can try to be more concise in our README

🤖 AI-assisted comment


### `InvocationOtelPluginFactory`

Factory for the invocation-rooted plugin, and what belongs in the SDK's `plugins`
Expand Down Expand Up @@ -577,7 +645,7 @@ picked up.

Factory for the execution-rooted plugin, with the same construction and config as
`InvocationOtelPluginFactory`. `EXECUTION_OTEL_PLUGIN_FACTORY` is the
default-configured instance the `otel-execution` entry point names.
default-configured instance both the `otel` and `otel-execution` entry points name.

### `InvocationOtelPlugin`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies = [
]

[project.entry-points."aws_durable_execution.plugins"]
otel = "aws_durable_execution_sdk_python_otel.plugin_factory:EXECUTION_OTEL_PLUGIN_FACTORY"
otel-invocation = "aws_durable_execution_sdk_python_otel.plugin_factory:INVOCATION_OTEL_PLUGIN_FACTORY"
otel-execution = "aws_durable_execution_sdk_python_otel.plugin_factory:EXECUTION_OTEL_PLUGIN_FACTORY"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
from aws_durable_execution_sdk_python_otel.plugin_factory import (
EXECUTION_OTEL_PLUGIN_FACTORY,
INVOCATION_OTEL_PLUGIN_FACTORY,
create_otel_plugin_factory,
ExecutionOtelPluginFactory,
InvocationOtelPluginFactory,
)
Expand All @@ -49,6 +50,7 @@
"ExecutionOtelPluginFactory",
"ExtractedContext",
"OtelPluginConfig",
"create_otel_plugin_factory",
Comment thread
zhongkechen marked this conversation as resolved.
Resolved
"InvocationOtelPlugin",
"InvocationOtelPluginFactory",
"OtelContextLogFilter",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

from __future__ import annotations

from typing import TYPE_CHECKING, ClassVar
from typing import TYPE_CHECKING, ClassVar, Literal, overload

from aws_durable_execution_sdk_python_otel.execution_plugin import (
ExecutionOtelPlugin,
Expand Down Expand Up @@ -94,8 +94,71 @@ def create_plugin(self, info: InvocationStartInfo) -> ExecutionOtelPlugin:
return ExecutionOtelPlugin(self.config)


@overload
def create_otel_plugin_factory(
config: OtelPluginConfig | None = None,
*,
mode: Literal["execution"] = "execution",
) -> ExecutionOtelPluginFactory: ...


@overload
def create_otel_plugin_factory(
config: OtelPluginConfig | None = None,
*,
mode: Literal["invocation"],
) -> InvocationOtelPluginFactory: ...


# A mode read from configuration at runtime is a plain ``str``, which neither
# literal overload accepts. This last overload keeps that call type-checkable;
# the union it returns is what a caller can know without a literal, and an
# unsupported value still raises ``ValueError`` below.
@overload
def create_otel_plugin_factory(
config: OtelPluginConfig | None = None,
*,
mode: str,
) -> ExecutionOtelPluginFactory | InvocationOtelPluginFactory: ...


def create_otel_plugin_factory(
config: OtelPluginConfig | None = None,
*,
mode: str = "execution",
) -> ExecutionOtelPluginFactory | InvocationOtelPluginFactory:
"""Return a new factory for the selected view, defaulting to execution.

Each call constructs a factory of the selected view's class, carrying that
view's ``exclusive_group`` and registration metadata. Calls are not
memoized, so register exactly one: two factories of the same view in
``plugins`` are rejected at registration as a duplicate in the
``aws-durable-execution-otel-view`` exclusive group. Configuration is passed
unchanged to that class. Omitted configuration uses the globally configured
OpenTelemetry provider, resolved per invocation rather than here.

Args:
config: Existing OTel plugin configuration.
mode: ``"execution"`` for Workflow-parented operations, or
``"invocation"`` for Invocation-parented operations.

Raises:
ValueError: If mode is neither ``"execution"`` nor ``"invocation"``.
"""
if mode == "execution":
return ExecutionOtelPluginFactory(config)
if mode == "invocation":
return InvocationOtelPluginFactory(config)
raise ValueError("mode must be 'execution' or 'invocation'")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: now that a runtime str is accepted, it would help to show the rejected value, e.g. raise ValueError(f"mode must be 'execution' or 'invocation', got {mode!r}").

🤖 AI-assisted comment



INVOCATION_OTEL_PLUGIN_FACTORY = InvocationOtelPluginFactory()
"""Default-configured factory named by the ``otel-invocation`` entry point."""

EXECUTION_OTEL_PLUGIN_FACTORY = ExecutionOtelPluginFactory()
"""Default-configured factory named by the ``otel-execution`` entry point."""
"""Default-configured factory named by the ``otel`` and ``otel-execution`` entry points.

Both names resolve to this one object, so ``DURABLE_EXECUTION_PLUGINS=otel,otel-execution``
loads a single factory: the loader skips the second by identity and logs a warning.
``otel`` with ``otel-invocation`` names two different views and is rejected.
"""
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,19 @@
from __future__ import annotations

import json
from collections.abc import Callable
from functools import partial
import logging
import threading
from dataclasses import replace
from datetime import UTC, datetime
from typing import Any
from unittest.mock import Mock, patch

import pytest

from aws_durable_execution_sdk_python.plugin import DurableInstrumentationPluginFactory
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory
from aws_durable_execution_sdk_python.context import DurableContext, durable_step
from aws_durable_execution_sdk_python.execution import (
InvocationStatus,
Expand All @@ -37,9 +43,6 @@
)
from aws_durable_execution_sdk_python_otel.log_filter import OtelContextLogFilter
from aws_durable_execution_sdk_python_otel.otel_plugin_config import OtelPluginConfig
from aws_durable_execution_sdk_python_otel.plugin_factory import (
InvocationOtelPluginFactory,
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
Expand Down Expand Up @@ -180,7 +183,17 @@ def _remove_otel_filters(handler: logging.Handler) -> None:
handler.removeFilter(installed)


def test_overlapping_invocations_stamp_top_level_logs_with_their_own_span() -> None:
@pytest.mark.parametrize(
("create_factory", "parent_name"),
[
(create_otel_plugin_factory, "Workflow"),
(partial(create_otel_plugin_factory, mode="invocation"), "Invocation"),
],
)
def test_overlapping_invocations_stamp_top_level_logs_with_their_own_span(
create_factory: Callable[[OtelPluginConfig], DurableInstrumentationPluginFactory],
parent_name: str,
) -> None:
"""A handler's first log carries its own invocation's trace and span ids.

Both invocations are open when either record is emitted, and neither record
Expand All @@ -201,7 +214,7 @@ def test_overlapping_invocations_stamp_top_level_logs_with_their_own_span() -> N
provider.add_span_processor(SimpleSpanProcessor(exporter))
# enrich_logger is the documented default: the plugin installs the filter on
# the root logger's handlers, including the collector added above.
factory = InvocationOtelPluginFactory(
factory = create_factory(
OtelPluginConfig(tracer_provider=provider, enrich_logger=True)
)

Expand Down Expand Up @@ -283,6 +296,20 @@ def invoke(owner: str) -> None:
assert stamped == {
owner: expected[_execution_arn(owner)] for owner in OWNERS
}, f"records carried {stamped}"
# Log correlation keeps its existing Invocation fallback in both views;
# durable operation parenting distinguishes the selected view.
parents = {
str(span.attributes["durable.execution.arn"]): span.context
for span in exporter.get_finished_spans()
if span.name == parent_name and span.attributes is not None
}
for owner in OWNERS:
operation = next(
span
for span in exporter.get_finished_spans()
if span.name == f"after-log-{owner}"
)
assert operation.parent == parents[_execution_arn(owner)]
finally:
for installed_on in list(root.handlers):
_remove_otel_filters(installed_on)
Expand Down
Loading
Loading