Skip to content

fix(otel): preserve the final polling failure without bypassing quiescence #170

Open

Description

@zhongkechen

After one acceptable trace is observed, subsequent retryable backend errors are reported as a generic stability timeout. The final backend error is hidden even though it would explain why stability could not be established.

Follow-up to PR #131's polling review comment.

Reproduction and impact

Validated with Python 3.14.9 at PR head 04f194f44a5136074dfb17c8add1d75b24055cad; the affected implementation remains present in main 9a5ead5e9ab8051bedfd300726472cbd443923cb on 2026-10-27.

Use the deterministic _TimedBackend and _one_span_trace() helpers from tests/test_polling.py:

  1. Return a passing trace on the first lookup.
  2. Raise RetryableBackendError("HTTP 429: backend unavailable") on the next two lookups.
  3. Set PollingPolicy(timeout_seconds=10, interval_seconds=1, max_attempts=3, quiescence_seconds=2).

The result is TelemetryTimeout("Correlated trace did not remain valid and unchanged for 2s within the polling budget"), with the observed trace preserved. The HTTP 429 message is lost. Replacing the final two lookups with None produces the same timeout.

The existing behavior fails validation safely. This issue is about preserving an actionable reason for that failure.

Important correction to the original suggestion

Do not fix this solely by setting latest_acceptable = False in the empty-result and retryable-error branches.

Applying those two resets in memory makes both reproductions return the earlier passing trace through return latest_trace at line 171. The extension validates that returned trace again and can report success even though no uninterrupted stability interval was confirmed.

Requested behavior / acceptance criteria

  • When polling ends on a retryable backend failure, surface that failure or include it in the timeout diagnostic.
  • Preserve the latest observed trace on the raised BackendError so failure artifacts remain available.
  • An acceptable trace followed only by empty results must still fail when quiescence was never established; identify the observation gap in the diagnostic.
  • Never return an earlier passing trace solely because an empty lookup or error reset the stability state.
  • Preserve the existing behavior of returning invalid observed telemetry for detailed validation errors where appropriate, and preserve zero-quiescence behavior.
  • Add deterministic tests for acceptable trace followed by exhausted errors, exhausted empty results, and recovery followed by a complete stability interval.

The reproduction is local harness logic and requires no SDK, AWS credentials, or network. The unchanged focused validation set passed all 277 tests, including test_polling.py; it did not cover these exhaustion sequences.

Activity

  1. added
    bugSomething isn't working
    pkg:otelPackage: aws-durable-execution-conformance-tests-otel
    and removed
    needs-triageAwaiting operator triage
    on Oct 28, 2026
  2. zhongkechen commented on Nov 2, 2026

    @zhongkechen
    ContributorAuthor

    Severity - low

    When it triggers. With a nonzero quiescence interval, polling first observes an acceptable trace, then exhausts its budget on retryable backend failures or empty lookups before stability is established.

    Impact. The harness reports a generic stability timeout and preserves the observed trace, but hides the final backend failure or observation gap. For example, an HTTP 429 is no longer visible in the terminal diagnostic, making a failed conformance run harder to investigate.

    What is correct. The current exhaustion path still fails validation when stability was never established. It does not turn the earlier acceptable trace into a passing result in the reported sequence, and it does not affect customer SDK execution or telemetry generation.

    Why this severity. The defect is limited to the explanation of an already failing test run; the validation decision and failure artifact are preserved. The warning about resetting latest_acceptable describes a hazard in a proposed fix, not a false-pass defect in the current implementation. Preserve that safety property while surfacing the final failure.

    Source review: polling.py at 0eaf7341, consistent with the deterministic reproduction in the issue. No new test run was performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpkg:otelPackage: aws-durable-execution-conformance-tests-otel

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions