Repository navigation
test: adopt shared LMI lifecycle suite - #931
Draft
zhongkechen wants to merge 41 commits into
Draft
zhongkechen wants to merge 41 commits into
zhongkechen wants to merge 41 commits into
Conversation
Add opt-in cloud deployment and lifecycle coverage for Python LMI. Assert invocation deadlines, early branch cleanup, replay, and worker recovery; keep issue 741 regressions red until the SDK fix.
The root only coordinates workspace packages. Disable its implicit editable install so the LMI test directory does not trigger setuptools' multiple top-level package discovery error when starting examples. Exercise examples environment initialization in the unprivileged LMI harness job and verify that it imports the checked-out SDK.
Run the complete cloud matrix on every trusted PR update, including drafts, and every main push without labels or path filters. Preserve existing cloud credential restrictions for forks and Dependabot. Use run-owned resources without a cross-run concurrency group so newer commits cannot replace another commit's pending cloud job.
Initialize explicit hold/release objects before invoking fixtures and report missing or forbidden controls as infrastructure failures. Signal the winner only after the losing step has entered held I/O, and reject cleanup assertions when that loser exited before the winner was ready. Poll events by case with bounded parallel reads, correlate checkpoint holds to their request, and check stale effects before waiting for a retry on a potentially pinned worker. Cover the observed false positives and full fixture handshakes with regression tests.
Keep LMI on Python 3.14 with environment concurrency one and two. Exercise root finally before PENDING, bounded nested progress, concurrent shared-handler invocations, in-flight return/failure cleanup, and late operations from abandoned children using Python public APIs. Document the Java/Python mapping and preserve the newly exposed late-step scope defect as an explicit issue 741 regression.
Collect each case's own executions during teardown and leave full-run history/log collection to the final workflow step. Pace history reads and bound throttle-only retries, including reads through the public cloud runner, without retrying invocation or callback writes. Cover pagination, throttling exhaustion, permission errors, and scoped versus full-run collection with focused tests.
Checkpoint a victim's placement decision before starting fault work. If a runtime environment rotates between anchor and victim admission, replace the pair with a bounded retry. Never retry an admitted lifecycle assertion. Keep the admission decision stable during replay.
Hold a repository-wide concurrency slot through deployment, testing, and cleanup without increasing shared provider capacity. Retain up to 100 pending cloud jobs with queue: max instead of evicting older jobs. Keep the harness parallel and document the queue and repository scope.
Deploy one function per native concurrency setting and reuse it for ordinary and deadline scenarios. Run all 28 cases in one cloud job without changing provider capacity or losing regression assertions. Release controls, stop remaining executions, and observe wrapper exit before reusing each function. Extend fault observation for the shared 60-second timeout and retain a bounded emergency release.
Create the persistent stack once and update the same two functions after previous durable executions finish. Preserve failed stacks, code artifacts, and infrastructure after every test run. Isolate controls and events by run ID, expire only run data, and scope cleanup to the current run without deleting functions or buckets. Remove the infrastructure janitor and verify create/update behavior.
Keep strict cloud regressions manual while the harness runs on PRs. Pin artifact uploads to the immutable revision used on current main. Invalidate old run metadata before build and deployment preflight, preserving the build manifest and preventing prior-run cleanup.
…lmi-observability-python-20261023
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adapts the Python SDK to the seven lifecycle requirements and reusable harness in conformance #186, pinned to
56e7e444457653775579c120469d931b71c8cfd8..github/workflows/lmi-e2e-tests.ymlcalls the sharedlmi-lifecycle.ymlat that exact SHA. The shared workflow owns harness/cloud jobs, AWS credentials and account checks, deployment/test budgets, collection, cleanup, artifacts and summary..github/scripts/lmi-ci.shsupplies Python setup/build/test commands. GitHub Actions' fixed cloud-job concurrency group remains the sole cross-run guard, covering deployment through cleanup; no S3 shared lock is used.Reusable AWS transport, deployment, scenario orchestration, evidence normalization, reporting, cleanup and their regression tests moved to #186. Python retains its real public SDK fixtures, lifecycle observation hooks, core wheel/ZIP build, bootstrap and thin runtime bindings. The Lambda bundle contains only the checked-out core SDK, its dependencies and fixture/observation modules. Production SDK behavior is unchanged. OTel/exporter/plugin conformance and the former Python-specific catalog are deferred.
All seven cases are enabled in their default order:
lmi-lifecycle-1native-overlaplmi-lifecycle-2normal-returnlmi-lifecycle-3deadline-during-steplmi-lifecycle-4peer-recoverylmi-lifecycle-5suspension-retrylmi-lifecycle-6warm-reuselmi-lifecycle-7normal-failureThe adapter reads the shared matrix directly. Identity is run/case based, without
cohort_idorcohortId. Probes after both wrapper return and raise preserve actual SDK admission behavior. Case 4 checkpoints the victim's original request ID, admits the peer halfway through the victim timeout, verifies the remaining recovery window and excludes expired originals or service retries from healthy-peer evidence. Retries do not reinject the victim fault. Completion timestamps are captured once in milliseconds and reused without conversion drift.Validation:
hatch run lmi:shared-tests: 61 passed, including real public-SDK local e2e tests for replay, normal return/failure, warm reuse and recovery instrumentation.hatch run lmi:lint, scoped mypy, Bash syntax and stage failure/identity tests: passed.These counts validate the harness, not deployed SDK conformance. Existing late-admission behavior remains visible as failure: an asynchronous checkpoint can be accepted after normal completion and a late step can remain pending. Earlier cloud diagnostic run 43527018694 passed case 4; diagnostic runs also passed cases 1 and 5 while cases 3, 6 and 7 retained findings. Whole-suite cloud acceptance is still pending.
See run instructions and suite ownership. Merge #186 before #931; retain the exact shared revision pin.