Repository navigation
feat(telemetry): add telemetry-2 failure handler - #907
Conversation
Add the handler for conformance requirement telemetry-2.
The step "charge" raises "card declined" with no retry.
The handler catches the StepError and raises "payment failed" from it.
So the execution fails with a three-level live exception chain.
Add the template function, the Makefile target, the wiring test updates,
and the README row.
The telemetry exporter writes every error as {name, message}. The plugin
hooks pass it an ErrorObject with no cause. So the Python record has no
cause at either level, and telemetry-2 fails for this SDK until the
exporter writes the chain. The README states this.
Requirement: aws/aws-durable-execution-conformance-tests#179
JS handler: aws/aws-durable-execution-sdk-js#1168
The coverage table row describes the requirement. The note about the exporter's missing cause chain is removed from the README; the PR description carries it.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
talked with team and the error cause depth difference between SDKs is expected. Need to check if README files need updating. For error tests just need to check for the fact that an error exists. Do not worry about cause depth and error names. |
This comment has been minimized.
This comment has been minimized.
The current pin d4b7df9 predates conformance PR #179, so the runner it installs has no telemetry-2 requirement and the suite filters the new handler out. Pin to the #179 head 31d4363 so this PR's run exercises telemetry-2. telemetry-suite.yml is identical between the two commits, so the input contract is unchanged. TEMPORARY: bump to the #179 merge commit on main before merging.
The team decided that error names and cause depth differ per SDK by design. Conformance PR #179 at 121e832 now matches each telemetry-2 `error` with `{$object: {}}`: present and an object, keys unchecked. So this SDK's `{name, message}` errors with no `cause` pass. The README row, the handler docstring, and the test docstring no longer describe a cause chain. The TEMPORARY workflow pin moves from the previous #179 head 31d4363 to 121e832. telemetry-suite.yml is identical between the two commits. Bump to the #179 merge commit on main before merging.
ecfe578 to
e1ce154
Compare
This comment has been minimized.
This comment has been minimized.
The two source-text tests asserted string literals. The handler has three `raise RuntimeError` statements, so deleting the log-group guard left the test green. Changing `except StepError` to `except ValueError` left every literal in place, so the handler could stop raising `payment failed` with no local failure. Rule 4 requires a test that fails when a branch is removed. Replace them with behavioral tests. Each module is executed from its file, as Lambda does. One test deletes TELEMETRY_LOG_GROUP_NAME and expects the import-time RuntimeError, for both modules. Two tests run each handler through DurableFunctionTestRunner with CloudWatchLogsExporter.export replaced by a capture of the rendered record. The telemetry-2 test asserts FAILED, `payment failed`, and a `charge` STEP with failedCount 1, maxAttempt 1 and `card declined`. This is the local run the PR description already reported, now committed. Checked by mutation: deleting the guard, changing the except type, and dropping RetryPresets.none() each fail exactly the covering test.
This comment has been minimized.
This comment has been minimized.
The handler constructs CloudWatchLogsExporter at import, and the
exporter calls boto3.client("logs") in its constructor. botocore
resolves credentials when it creates a client (botocore/session.py,
create_client calls get_credentials). So the previous tests ran the
credential chain, including IMDS, on every handler import, and a bad
AWS_PROFILE or a proxy could fail them before the handler ran.
Replace boto3.client with a FakeLogsClient for the handler tests. The
fake records each put_log_events call. The tests now run the real
export and render path and read the record from the log event message,
so they also check the log group name reaches the exporter. The export
monkeypatch and the AWS_DEFAULT_REGION setting are removed.
Verified with AWS_PROFILE=does-not-exist, AWS_EC2_METADATA_DISABLED,
no region and empty config files: 17 passed. The three mutation checks
from the previous commit still fail exactly the covering test.
This comment has been minimized.
This comment has been minimized.
ParidelPooya
left a comment
There was a problem hiding this comment.
The handler matches the JS handler, and the tests check behavior: each mutation listed in the PR body fails exactly one test. CI passes telemetry-2. One item must change before merge.
Blocking
.github/workflows/telemetry-conformance-tests.yml:48pins the shared workflow to121e832. That commit is only on the branch of aws/aws-durable-execution-conformance-tests#179, which is still open. The conformance repository deletes the head branch on merge. So after #179 merges,121e832is on no branch. The comment on line 43 says the pinned SHA must be reachable from a branch. Fix: after #179 merges, pin to its merge commit on conformancemainand restore the permanent comment.
Non-blocking
packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/README.md:65-66says the tests run each handler with the exporterexportcaptured. Commita6055bb6replaced that capture with a fakeboto3.client(tests/test_telemetry_examples.py:257). So the tests now run the realexportpath. Suggestion: "withboto3.clientreplaced by a fakelogsclient that recordsPutLogEventscalls".- The docstring at
tests/test_telemetry_examples.py:4-10says the tests pin the template wiring. The module now also runs both handlers throughDurableFunctionTestRunnerand checks the exported record. Suggestion: add one sentence that says so. - The message of commit
1911c787saystelemetry-2fails for this SDK. Commite1ce1544reverses this. No action is needed if the PR is squash-merged. - Outside this PR:
mypyreportsToo few arguments [call-arg]ondurable_execution(plugins=...)atsrc/telemetry_2_failure.py:57.mainhas the same error atsrc/telemetry_1_success.py:42.durable_executionhas no@overloadfor the factory form. CI does not type-check this package, so nothing fails. A separate change could add the overloads.
The README said the tests capture the exporter's export. The tests now replace boto3.client with a fake logs client that records PutLogEvents calls. Update the README sentence and the test module docstring to match.
This comment has been minimized.
This comment has been minimized.
|
Pushed Item 2. Item 3. The module docstring at Item 1 (blocking). The workflow pin stays at Item 4. No change. The PR is squash-merged, so the message of Item 5. Out of scope for this PR. Verification: |
Codex AI reviewNo actionable findings across the six changed files after checking correctness and CONTRIBUTING rules 1–9. Prior findings are addressed. Residual risk: tests and cloud conformance were not rerun, and the external workflow pin was not independently verified. Reviewed commit |
ParidelPooya
left a comment
There was a problem hiding this comment.
Thanks. I checked b2dcc9d. All items are addressed.
- The workflow pins the shared workflow to
ca2cd02, which is the #179 merge commit on conformancemain. The telemetry job passes on this pin. - The README now describes the fake
boto3.clientboundary. - The test module docstring now says the module also runs both handlers and checks the record.
Approving.
Summary
Adds the Python handler for conformance requirement
telemetry-2(aws/aws-durable-execution-conformance-tests#179).
The JS handler is aws/aws-durable-execution-sdk-js#1168.
The step
chargeraisescard declined. The retry strategy isRetryPresets.none(), sothe step fails on its first attempt. The handler catches the
StepErrorand raisespayment failedfrom it. So the execution fails and the live exception chain has threelevels: the handler error, the
StepError, and theDurableOperationErrorthe SDK rebuiltfrom the checkpointed step error.
Design
Problem. The telemetry conformance suite gains requirement telemetry-2: a failed
single-step execution must produce one telemetry record with status
FAILED, an executionerror, and a
chargeSTEP operation withfailedCount 1andmaxAttempt 1. This packagehas no handler that produces that record.
Cause. The package has one handler,
telemetry_1_success, and it succeeds. The suitematches a deployed function to a requirement by name, so each requirement needs a function
that produces its record.
Options considered.
telemetry_1_successselects success or failure. Rejected. Thesuite maps one function to one requirement by function name, so one module would still
need two template functions. The flag adds a branch to the handler that would itself
need a test (rule 4). The JS (#1168) and Java (chore: release SDK 3.0.1 #876) packages use one module per
requirement.
operation record.
maxAttempt 1, and thedefault preset retries the step.
StepErrorpropagate unwrapped. Rejected. telemetry-2 pins the executionerror message
payment failedinExpectedResult, and theStepErrormessage is thestep's message.
_error_shapein the exporter to emit acausechain. Rejected. The pluginhooks receive an
ErrorObjectwith type, message, data and stack trace, and no cause(
telemetry-exporter/.../plugin.py:386–394). The team accepted that cause depth anderror names differ per SDK. docs(sdk): add serialization docs #179 at
121e832matches eacherrorwith{$object: {}}.telemetry_2_failure.py, with the same pluginconfiguration as telemetry-1. One step with
RetryPresets.none()that raisescard declined; the handler catchesStepErrorand raisespayment failedfrom it.Reason: it is the smallest change that produces the required record, it adds no branch
to existing code, and it matches the per-requirement layout of the other SDKs.
Removed. Nothing.
test_coverage_is_exactly_telemetry_1and the single-functionsettings test are replaced by parametrized versions over both functions.
Decisions not covered by #179
RuntimeError. docs(sdk): add serialization docs #179 checks the messages only, so the type isfree;
RuntimeErrormatches the JS handler's plainError.ca2cd02on main. See "Workflow pin" below.boto3.clientwith a fakelogsclient that recordsput_log_eventscalls. The handler constructs the exporter itself at import, andbotocore resolves credentials when it creates a client, so the fake is the boundary that
keeps the tests off the credential chain while running the real
exportandrender.Changes
packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/src/telemetry_2_failure.py: new handler, same plugin config as telemetry-1.src/Makefile:build-Telemetry2Failuretarget. Both targets share one recipe.template.yaml:Telemetry2Failurefunction mapped totelemetry-2.tests/test_telemetry_examples.py: covers both functions in the template tests. Runsboth handler modules locally: a test per module that unsets
TELEMETRY_LOG_GROUP_NAMEand expects the import-time
RuntimeError; a test per handler that runs it throughDurableFunctionTestRunnerwith a fake boto3logsclient and asserts the JSON recordin the
PutLogEventsmessage and the log group name. The telemetry-2 test assertsFAILED,payment failed, and achargeSTEP withfailedCount 1,maxAttempt 1,card declined.README.md: coverage row, layout line, test description.Error shape
The plugin hooks pass the exporter an
ErrorObjectwith type, message, data and stack trace,and no cause. So
_error_shapeintelemetry-exporter/.../plugin.pywrites every error as{name, message}. The JS SDK writes a three-level and a two-level chain, the Java SDK two andone. The team decided this difference is expected. telemetry-2 at conformance commit
121e832matches each
errorwith{$object: {}}: present and an object, keys unchecked. The messagesare asserted by the core
ExpectedResultandExpectedExecutionHistorysections. So this SDKpasses telemetry-2 as is.
Workflow pin
telemetry-conformance-tests.ymlpins the shared workflow toca2cd02, the #179 merge commiton main. Earlier revisions of this PR pinned the #179 head
121e832so the run exercisedtelemetry-2 before #179 merged.
telemetry-suite.ymlis identical to the previous pin.Verification
121e832(run 41892370013, samehandler and template as the current head):
telemetry-1 PASSED,telemetry-2 PASSED.hatch run test:all packages/aws-durable-execution-sdk-python-conformance-tests-telemetry/tests: 17 passed.Also 17 passed with
AWS_PROFILE=does-not-exist,AWS_EC2_METADATA_DISABLED=true, no regionand empty config files, so the tests do not depend on credential discovery.
hatch fmt --checkin the package: ruff check and format clean.telemetry_2_failure.py, each run against the committed tests:TELEMETRY_LOG_GROUP_NAMEguard →test_handler_module_fails_fast_without_log_group[telemetry_2_failure]fails, 16 pass;except StepError→except ValueError→test_telemetry_2_handler_fails_with_payment_failed_after_one_attemptfails on'card declined' == 'payment failed';RetryPresets.none()→ the same test fails onmaxAttempt.