Skip to content

test: adopt shared LMI lifecycle suite - #931

Draft
zhongkechen wants to merge 41 commits into
mainfrom
feat/python-lmi-shared-suite
Draft

zhongkechen wants to merge 41 commits into
mainfrom
feat/python-lmi-shared-suite

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This adapts the Python SDK to the seven lifecycle requirements and reusable harness in conformance #186, pinned to 56e7e444457653775579c120469d931b71c8cfd8.

.github/workflows/lmi-e2e-tests.yml calls the shared lmi-lifecycle.yml at that exact SHA. The shared workflow owns harness/cloud jobs, AWS credentials and account checks, deployment/test budgets, collection, cleanup, artifacts and summary. .github/scripts/lmi-ci.sh supplies Python setup/build/test commands. GitHub Actions' fixed cloud-job concurrency group remains the sole cross-run guard, covering deployment through cleanup; no S3 shared lock is used.

Reusable AWS transport, deployment, scenario orchestration, evidence normalization, reporting, cleanup and their regression tests moved to #186. Python retains its real public SDK fixtures, lifecycle observation hooks, core wheel/ZIP build, bootstrap and thin runtime bindings. The Lambda bundle contains only the checked-out core SDK, its dependencies and fixture/observation modules. Production SDK behavior is unchanged. OTel/exporter/plugin conformance and the former Python-specific catalog are deferred.

All seven cases are enabled in their default order:

Requirement Scenario Native target Roles
lmi-lifecycle-1 native-overlap c2 victim, peer
lmi-lifecycle-2 normal-return c1 victim
lmi-lifecycle-3 deadline-during-step c1 victim
lmi-lifecycle-4 peer-recovery c2 victim, peer, recovery
lmi-lifecycle-5 suspension-retry c1 victim, resume
lmi-lifecycle-6 warm-reuse c1 victim, warm
lmi-lifecycle-7 normal-failure c1 victim

The adapter reads the shared matrix directly. Identity is run/case based, without cohort_id or cohortId. Probes after both wrapper return and raise preserve actual SDK admission behavior. Case 4 checkpoints the victim's original request ID, admits the peer halfway through the victim timeout, verifies the remaining recovery window and excludes expired originals or service retries from healthy-peer evidence. Retries do not reinject the victim fault. Completion timestamps are captured once in milliseconds and reused without conversion drift.

Validation:

  • hatch run lmi:shared-tests: 61 passed, including real public-SDK local e2e tests for replay, normal return/failure, warm reuse and recovery instrumentation.
  • hatch run lmi:lint, scoped mypy, Bash syntax and stage failure/identity tests: passed.
  • Exact pinned checkout/bootstrap/import and workflow reference verification: passed.
  • Python 3.14 arm64 Lambda bundle built and inspected: core SDK/dependencies plus three fixture files; no OTel or conformance package.
  • Shared chore(sdk): update supported python version #186 workspace: 1,923 tests passed, including the migrated driver tests, with full types/lint and distribution validation.

These counts validate the harness, not deployed SDK conformance. Existing late-admission behavior remains visible as failure: an asynchronous checkpoint can be accepted after normal completion and a late step can remain pending. Earlier cloud diagnostic run 43527018694 passed case 4; diagnostic runs also passed cases 1 and 5 while cases 3, 6 and 7 retained findings. Whole-suite cloud acceptance is still pending.

See run instructions and suite ownership. Merge #186 before #931; retain the exact shared revision pin.

zhongkechen and others added 23 commits September 23, 2026 00:24
Add opt-in cloud deployment and lifecycle coverage for Python LMI.
Assert invocation deadlines, early branch cleanup, replay, and worker
recovery; keep issue 741 regressions red until the SDK fix.
The root only coordinates workspace packages. Disable its implicit
editable install so the LMI test directory does not trigger setuptools'
multiple top-level package discovery error when starting examples.

Exercise examples environment initialization in the unprivileged LMI
harness job and verify that it imports the checked-out SDK.
Run the complete cloud matrix on every trusted PR update, including
drafts, and every main push without labels or path filters. Preserve
existing cloud credential restrictions for forks and Dependabot.

Use run-owned resources without a cross-run concurrency group so newer
commits cannot replace another commit's pending cloud job.
Initialize explicit hold/release objects before invoking fixtures and
report missing or forbidden controls as infrastructure failures. Signal
the winner only after the losing step has entered held I/O, and reject
cleanup assertions when that loser exited before the winner was ready.

Poll events by case with bounded parallel reads, correlate checkpoint
holds to their request, and check stale effects before waiting for a
retry on a potentially pinned worker. Cover the observed false positives
and full fixture handshakes with regression tests.
Keep LMI on Python 3.14 with environment concurrency one and two.
Exercise root finally before PENDING, bounded nested progress, concurrent
shared-handler invocations, in-flight return/failure cleanup, and late
operations from abandoned children using Python public APIs.

Document the Java/Python mapping and preserve the newly exposed late-step
scope defect as an explicit issue 741 regression.
Collect each case's own executions during teardown and leave full-run
history/log collection to the final workflow step. Pace history reads
and bound throttle-only retries, including reads through the public
cloud runner, without retrying invocation or callback writes.

Cover pagination, throttling exhaustion, permission errors, and scoped
versus full-run collection with focused tests.
Checkpoint a victim's placement decision before starting fault work.
If a runtime environment rotates between anchor and victim admission,
replace the pair with a bounded retry. Never retry an admitted lifecycle
assertion. Keep the admission decision stable during replay.
Hold a repository-wide concurrency slot through deployment, testing,
and cleanup without increasing shared provider capacity. Retain up to
100 pending cloud jobs with queue: max instead of evicting older jobs.

Keep the harness parallel and document the queue and repository scope.
Deploy one function per native concurrency setting and reuse it for
ordinary and deadline scenarios. Run all 28 cases in one cloud job
without changing provider capacity or losing regression assertions.

Release controls, stop remaining executions, and observe wrapper exit
before reusing each function. Extend fault observation for the shared
60-second timeout and retain a bounded emergency release.
Create the persistent stack once and update the same two functions
after previous durable executions finish. Preserve failed stacks,
code artifacts, and infrastructure after every test run.

Isolate controls and events by run ID, expire only run data, and scope
cleanup to the current run without deleting functions or buckets.
Remove the infrastructure janitor and verify create/update behavior.
Keep strict cloud regressions manual while the harness runs on PRs.
Pin artifact uploads to the immutable revision used on current main.
Invalidate old run metadata before build and deployment preflight,
preserving the build manifest and preventing prior-run cleanup.
@zhongkechen
zhongkechen deleted the feat/python-lmi-shared-suite branch November 12, 2026 22:21
@zhongkechen
zhongkechen restored the feat/python-lmi-shared-suite branch November 12, 2026 22:27
@zhongkechen zhongkechen reopened this Nov 12, 2026
@zhongkechen zhongkechen changed the title test: adapt Python LMI fixtures to shared conformance test: run Python LMI fixtures through shared conformance Nov 12, 2026
@zhongkechen zhongkechen changed the title test: run Python LMI fixtures through shared conformance test: add Python-owned LMI regression coverage Nov 13, 2026
@zhongkechen zhongkechen changed the title test: add Python-owned LMI regression coverage test: adopt shared LMI lifecycle suite Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant