Skip to content

feat(telemetry): add telemetry-2 failure handler - #907

Merged
hln33 merged 8 commits into
mainfrom
feat/telemetry-2-failure-handler
Nov 5, 2026
Merged

hln33 merged 8 commits into
mainfrom
feat/telemetry-2-failure-handler

Conversation

@hln33

@hln33 hln33 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the Python handler for conformance requirement telemetry-2
(aws/aws-durable-execution-conformance-tests#179).
The JS handler is aws/aws-durable-execution-sdk-js#1168.

The step charge raises card declined. The retry strategy is RetryPresets.none(), so
the step fails on its first attempt. The handler catches the StepError and raises
payment failed from it. So the execution fails and the live exception chain has three
levels: the handler error, the StepError, and the DurableOperationError the SDK rebuilt
from the checkpointed step error.

Design

Problem. The telemetry conformance suite gains requirement telemetry-2: a failed
single-step execution must produce one telemetry record with status FAILED, an execution
error, and a charge STEP operation with failedCount 1 and maxAttempt 1. This package
has no handler that produces that record.

Cause. The package has one handler, telemetry_1_success, and it succeeds. The suite
matches a deployed function to a requirement by name, so each requirement needs a function
that produces its record.

Options considered.

  1. One input flag on telemetry_1_success selects success or failure. Rejected. The
    suite maps one function to one requirement by function name, so one module would still
    need two template functions. The flag adds a branch to the handler that would itself
    need a test (rule 4). The JS (#1168) and Java (chore: release SDK 3.0.1 #876) packages use one module per
    requirement.
  2. Raise from the handler body with no step. Rejected. telemetry-2 requires a failed STEP
    operation record.
  3. Keep the default retry preset. Rejected. telemetry-2 asserts maxAttempt 1, and the
    default preset retries the step.
  4. Let the StepError propagate unwrapped. Rejected. telemetry-2 pins the execution
    error message payment failed in ExpectedResult, and the StepError message is the
    step's message.
  5. Extend _error_shape in the exporter to emit a cause chain. Rejected. The plugin
    hooks receive an ErrorObject with type, message, data and stack trace, and no cause
    (telemetry-exporter/.../plugin.py:386–394). The team accepted that cause depth and
    error names differ per SDK. docs(sdk): add serialization docs #179 at 121e832 matches each error with {$object: {}}.
  6. Chosen. A second module, telemetry_2_failure.py, with the same plugin
    configuration as telemetry-1. One step with RetryPresets.none() that raises
    card declined; the handler catches StepError and raises payment failed from it.
    Reason: it is the smallest change that produces the required record, it adds no branch
    to existing code, and it matches the per-requirement layout of the other SDKs.

Removed. Nothing. test_coverage_is_exactly_telemetry_1 and the single-function
settings test are replaced by parametrized versions over both functions.

Decisions not covered by #179

  • Both raised errors are RuntimeError. docs(sdk): add serialization docs #179 checks the messages only, so the type is
    free; RuntimeError matches the JS handler's plain Error.
  • The workflow pin is the docs(sdk): add serialization docs #179 merge commit ca2cd02 on main. See "Workflow pin" below.
  • Local handler tests replace boto3.client with a fake logs client that records
    put_log_events calls. The handler constructs the exporter itself at import, and
    botocore resolves credentials when it creates a client, so the fake is the boundary that
    keeps the tests off the credential chain while running the real export and render.

Changes

packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/

  • src/telemetry_2_failure.py: new handler, same plugin config as telemetry-1.
  • src/Makefile: build-Telemetry2Failure target. Both targets share one recipe.
  • template.yaml: Telemetry2Failure function mapped to telemetry-2.
  • tests/test_telemetry_examples.py: covers both functions in the template tests. Runs
    both handler modules locally: a test per module that unsets TELEMETRY_LOG_GROUP_NAME
    and expects the import-time RuntimeError; a test per handler that runs it through
    DurableFunctionTestRunner with a fake boto3 logs client and asserts the JSON record
    in the PutLogEvents message and the log group name. The telemetry-2 test asserts FAILED, payment failed, and a
    charge STEP with failedCount 1, maxAttempt 1, card declined.
  • README.md: coverage row, layout line, test description.

Error shape

The plugin hooks pass the exporter an ErrorObject with type, message, data and stack trace,
and no cause. So _error_shape in telemetry-exporter/.../plugin.py writes every error as
{name, message}. The JS SDK writes a three-level and a two-level chain, the Java SDK two and
one. The team decided this difference is expected. telemetry-2 at conformance commit 121e832
matches each error with {$object: {}}: present and an object, keys unchecked. The messages
are asserted by the core ExpectedResult and ExpectedExecutionHistory sections. So this SDK
passes telemetry-2 as is.

Workflow pin

telemetry-conformance-tests.yml pins the shared workflow to ca2cd02, the #179 merge commit
on main. Earlier revisions of this PR pinned the #179 head 121e832 so the run exercised
telemetry-2 before #179 merged. telemetry-suite.yml is identical to the previous pin.

Verification

  • Telemetry Conformance Tests workflow at conformance pin 121e832 (run 41892370013, same
    handler and template as the current head): telemetry-1 PASSED, telemetry-2 PASSED.
  • hatch run test:all packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/tests: 17 passed.
    Also 17 passed with AWS_PROFILE=does-not-exist, AWS_EC2_METADATA_DISABLED=true, no region
    and empty config files, so the tests do not depend on credential discovery.
  • hatch fmt --check in the package: ruff check and format clean.
  • Rule 4 check by mutation on telemetry_2_failure.py, each run against the committed tests:
    • delete the TELEMETRY_LOG_GROUP_NAME guard → test_handler_module_fails_fast_without_log_group[telemetry_2_failure] fails, 16 pass;
    • except StepError → except ValueError → test_telemetry_2_handler_fails_with_payment_failed_after_one_attempt fails on 'card declined' == 'payment failed';
    • drop RetryPresets.none() → the same test fails on maxAttempt.

hln33 added 2 commits November 3, 2026 15:16
Add the handler for conformance requirement telemetry-2.
The step "charge" raises "card declined" with no retry.
The handler catches the StepError and raises "payment failed" from it.
So the execution fails with a three-level live exception chain.

Add the template function, the Makefile target, the wiring test updates,
and the README row.

The telemetry exporter writes every error as {name, message}. The plugin
hooks pass it an ErrorObject with no cause. So the Python record has no
cause at either level, and telemetry-2 fails for this SDK until the
exporter writes the chain. The README states this.

Requirement: aws/aws-durable-execution-conformance-tests#179
JS handler: aws/aws-durable-execution-sdk-js#1168
The coverage table row describes the requirement. The note about the
exporter's missing cause chain is removed from the README; the PR
description carries it.
@hln33
hln33 marked this pull request as ready for review November 3, 2026 23:02
@hln33
hln33 deployed to ai-pr-review-runtime November 3, 2026 23:02 — with GitHub Actions Active
@hln33
hln33 deployed to ai-pr-review-runtime November 3, 2026 23:03 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@hln33

hln33 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

talked with team and the error cause depth difference between SDKs is expected. Need to check if README files need updating.

For error tests just need to check for the fact that an error exists. Do not worry about cause depth and error names.

@github-actions

This comment has been minimized.

hln33 added 2 commits November 4, 2026 12:04
The current pin d4b7df9 predates conformance PR #179, so the runner
it installs has no telemetry-2 requirement and the suite filters the
new handler out. Pin to the #179 head 31d4363 so this PR's run
exercises telemetry-2. telemetry-suite.yml is identical between the
two commits, so the input contract is unchanged.

TEMPORARY: bump to the #179 merge commit on main before merging.
The team decided that error names and cause depth differ per SDK by
design. Conformance PR #179 at 121e832 now matches each telemetry-2
`error` with `{$object: {}}`: present and an object, keys unchecked.
So this SDK's `{name, message}` errors with no `cause` pass. The
README row, the handler docstring, and the test docstring no longer
describe a cause chain.

The TEMPORARY workflow pin moves from the previous #179 head 31d4363
to 121e832. telemetry-suite.yml is identical between the two commits.
Bump to the #179 merge commit on main before merging.
@hln33
hln33 force-pushed the feat/telemetry-2-failure-handler branch from ecfe578 to e1ce154 Compare November 4, 2026 19:05
@hln33
hln33 deployed to ai-pr-review-runtime November 4, 2026 19:06 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The two source-text tests asserted string literals. The handler has
three `raise RuntimeError` statements, so deleting the log-group guard
left the test green. Changing `except StepError` to `except ValueError`
left every literal in place, so the handler could stop raising
`payment failed` with no local failure. Rule 4 requires a test that
fails when a branch is removed.

Replace them with behavioral tests. Each module is executed from its
file, as Lambda does. One test deletes TELEMETRY_LOG_GROUP_NAME and
expects the import-time RuntimeError, for both modules. Two tests run
each handler through DurableFunctionTestRunner with
CloudWatchLogsExporter.export replaced by a capture of the rendered
record. The telemetry-2 test asserts FAILED, `payment failed`, and a
`charge` STEP with failedCount 1, maxAttempt 1 and `card declined`.
This is the local run the PR description already reported, now
committed.

Checked by mutation: deleting the guard, changing the except type, and
dropping RetryPresets.none() each fail exactly the covering test.
@hln33
hln33 deployed to ai-pr-review-runtime November 4, 2026 21:15 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The handler constructs CloudWatchLogsExporter at import, and the
exporter calls boto3.client("logs") in its constructor. botocore
resolves credentials when it creates a client (botocore/session.py,
create_client calls get_credentials). So the previous tests ran the
credential chain, including IMDS, on every handler import, and a bad
AWS_PROFILE or a proxy could fail them before the handler ran.

Replace boto3.client with a FakeLogsClient for the handler tests. The
fake records each put_log_events call. The tests now run the real
export and render path and read the record from the log event message,
so they also check the log group name reaches the exporter. The export
monkeypatch and the AWS_DEFAULT_REGION setting are removed.

Verified with AWS_PROFILE=does-not-exist, AWS_EC2_METADATA_DISABLED,
no region and empty config files: 17 passed. The three mutation checks
from the previous commit still fail exactly the covering test.
@hln33
hln33 deployed to ai-pr-review-runtime November 4, 2026 21:25 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The handler matches the JS handler, and the tests check behavior: each mutation listed in the PR body fails exactly one test. CI passes telemetry-2. One item must change before merge.

Blocking

  1. .github/workflows/telemetry-conformance-tests.yml:48 pins the shared workflow to 121e832. That commit is only on the branch of aws/aws-durable-execution-conformance-tests#179, which is still open. The conformance repository deletes the head branch on merge. So after #179 merges, 121e832 is on no branch. The comment on line 43 says the pinned SHA must be reachable from a branch. Fix: after #179 merges, pin to its merge commit on conformance main and restore the permanent comment.

Non-blocking

  1. packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/README.md:65-66 says the tests run each handler with the exporter export captured. Commit a6055bb6 replaced that capture with a fake boto3.client (tests/test_telemetry_examples.py:257). So the tests now run the real export path. Suggestion: "with boto3.client replaced by a fake logs client that records PutLogEvents calls".
  2. The docstring at tests/test_telemetry_examples.py:4-10 says the tests pin the template wiring. The module now also runs both handlers through DurableFunctionTestRunner and checks the exported record. Suggestion: add one sentence that says so.
  3. The message of commit 1911c787 says telemetry-2 fails for this SDK. Commit e1ce1544 reverses this. No action is needed if the PR is squash-merged.
  4. Outside this PR: mypy reports Too few arguments [call-arg] on durable_execution(plugins=...) at src/telemetry_2_failure.py:57. main has the same error at src/telemetry_1_success.py:42. durable_execution has no @overload for the factory form. CI does not type-check this package, so nothing fails. A separate change could add the overloads.

The README said the tests capture the exporter's export. The tests now
replace boto3.client with a fake logs client that records PutLogEvents
calls. Update the README sentence and the test module docstring to
match.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 17:57 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@hln33

hln33 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 72d372f8.

Item 2. packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/README.md:64-67 now says the tests run each handler through DurableFunctionTestRunner with boto3.client replaced by a fake logs client that records PutLogEvents calls.

Item 3. The module docstring at tests/test_telemetry_examples.py:9-10 now also says the module runs both handlers through DurableFunctionTestRunner and checks the exported record.

Item 1 (blocking). The workflow pin stays at 121e832 until aws/aws-durable-execution-conformance-tests#179 merges. It is then bumped to the merge commit and the permanent comment is restored.

Item 4. No change. The PR is squash-merged, so the message of 1911c787 does not reach main.

Item 5. Out of scope for this PR.

Verification: hatch run test:all on the package tests: 17 passed. hatch fmt --check: all checks passed.

The previous pin was the head of aws-durable-execution-conformance-tests
PR #179 (121e832), a temporary pin so the run exercised telemetry-2 before
#179 merged. #179 is merged as ca2cd02 on that repository's main branch, so
the pin now points at that merge commit.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 21:05 — with GitHub Actions Active
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings across the six changed files after checking correctness and CONTRIBUTING rules 1–9. Prior findings are addressed. Residual risk: tests and cloud conformance were not rerun, and the external workflow pin was not independently verified.

Reviewed commit b2dcc9d563ff2ffed4cdb3e4b64b82b934c3e733. Workflow run

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I checked b2dcc9d. All items are addressed.

  1. The workflow pins the shared workflow to ca2cd02, which is the #179 merge commit on conformance main. The telemetry job passes on this pin.
  2. The README now describes the fake boto3.client boundary.
  3. The test module docstring now says the module also runs both handlers and checks the record.

Approving.

@hln33
hln33 merged commit ae4ca53 into main Nov 5, 2026
22 checks passed
@hln33
hln33 deleted the feat/telemetry-2-failure-handler branch November 5, 2026 22:28

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — b2dcc9d5 Deployed Nov 5, 2026 by hln33 via ai-pr-review / Codex review / Generate Codex review #2859
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants