Skip to content

bug(e2e): Stabilize nondeterministic fixture request assertions in device-auth-health and openclaw-inference-switch #8002

Description

@sandl99

Investigation Summary

  • device-auth-health and openclaw-inference-switch synchronously inspect fixture request ledgers for /v1/chat/completions traffic that the tests do not explicitly trigger or await.
  • The same assertions pass and fail across commits that do not change the relevant tests or runtime behavior.
  • Failed runs still show successful setup/onboarding and authenticated /v1/models traffic; the expected chat-completions entry is absent when the ledger is read.
  • openclaw-inference-switch fails before exercising the switch, so the failure does not establish that inference switching is broken.
  • The exact source of the asynchronous request timing remains unknown.

Description

The live E2E targets device-auth-health and openclaw-inference-switch contain nondeterministic fixture request assertions. Each test assumes an incidental /v1/chat/completions request has already reached its fake provider, then immediately reads the fixture ledger.

When startup or onboarding has only issued /v1/models probes by that point, the assertion fails even though setup succeeds. The tests should explicitly produce and await the request required by their contract, or wait for a bounded deterministic fixture condition before asserting.

Observed sequence for openclaw-inference-switch:

For device-auth-health, the assertion passed in run 30614075121 and failed in run 30628616288 with the same class of missing incidental fixture request:

Expected: fixture assertions depend on requests explicitly initiated by the test and complete deterministically.

Actual: assertions depend on asynchronous startup/onboarding traffic being present at the instant the fixture ledger is read.

Reproduction Steps

  1. Dispatch the trusted E2E workflow with the device-auth-health and openclaw-inference-switch targets.
  2. Repeat the targets without changing their source.
  3. Inspect the failed fixture ledger.
  4. Observe that setup/onboarding completes and /v1/models is recorded, while the asserted /v1/chat/completions entry is sometimes absent.

Environment

  • GitHub-hosted Ubuntu E2E runners
  • OpenShell 0.0.85
  • OpenClaw 2026.7.1
  • NemoClaw commits and run receipts listed above

Debug Output

CI logs and uploaded E2E artifacts are attached to the linked workflow runs. The relevant failure is the fixture-ledger assertion expecting /v1/chat/completions after only /v1/models requests were recorded.

Acceptance Criteria

  • Both tests explicitly trigger the request they assert, or wait on a bounded fixture condition tied to the intended behavior.
  • A timeout reports the requests observed and the phase that failed.
  • Focused repeated runs demonstrate stable results without relying on incidental startup timing.

Checklist

  • I confirmed this bug is reproducible
  • I searched existing issues and this is not a duplicate

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: ciCI workflows, checks, release automation, or GitHub Actionsarea: e2eEnd-to-end tests, nightly failures, or validation infrastructureintegration: openclawOpenClaw integration behavior

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions