Repository navigation
feat(extras): rely on MicroVM auto-resume - #913
Merged
Merged
Conversation
The service resumes a suspended MicroVM on the next request when its idle policy sets autoResumeEnabled. So the SDK no longer resumes a MicroVM itself. - Remove ensure_running, its poll loop and EnsureRunningOptions. - Rename resume_if_suspended to check_not_ended. It makes one GetMicrovm call after a 502 MICROVM_CONNECT_FAILED or a request without an answer. TERMINATING, TERMINATED and not found fail at once. A denied call fails at once. Any other state keeps the job retrying. Any other error logs one warning per step attempt. - Remove MicrovmResumeDeniedError and the ResumeMicrovm call. - Stop writing resumeBeforeDelivery in the launch checkpoint. from_dict ignores the field, so an old checkpoint still replays. - Remove the in-memory suspend mark of a session handle and the recheck path of send_job. The test fake now fails any test that calls ResumeMicrovm.
A session that suspends its MicroVM must set an idle policy with autoResumeEnabled. The README now says so, with the idle timer rule for maxIdleDurationSeconds, the failure without the policy, the extra second of the first job after a suspend, and the replay of a launch checkpoint from the earlier release. - Remove lambda:ResumeMicrovm from the permissions. - Add a section on the state check, and remove the in-memory suspend record and the resume steps. - Update the worker README and one worker docstring: the service resumes the MicroVM, not the durable function.
The SDK no longer resumes a suspended MicroVM. So the two examples that suspend their MicroVM set an idle policy with autoResumeEnabled, a maxIdleDurationSeconds above the MicroVM's maximum duration, and a suspendedDurationSeconds of the maximum duration. The cloud tests keep their SUSPENDED-between-jobs and file-kept assertions. They now expect a launch checkpoint without resumeBeforeDelivery. The examples test README drops lambda:ResumeMicrovm from the role needs.
The rebase onto the dead-MicroVM fix keeps its order: after a sign, the state check runs first, and only the invocation reserve limits it. The check no longer waits for a resume. So the test of the resume-wait cap now proves that an attempt without a deadline ends, and the denial test covers only GetMicrovm. The worker README now says that the service resumes the MicroVM after a 503.
- Say that a job sent right after a suspend can take up to about 10 seconds. A retry_window under about 9 seconds relies on the step retry strategy for the rest of the 5-second window. - Add a suspend_after session without an idle_policy to the in-flight note. - Place the 30-second worker refusal after the suspend. - Use the Python names idle_policy and True, and the session timeout. - Add the suspendedDurationSeconds guideline. Its end is not verified. - Say that the service resumes the MicroVM in the worker docs. - Rewrap a long docstring line in an example.
ParidelPooya
marked this pull request as ready for review
November 7, 2026 13:19
Contributor
Codex AI reviewNo actionable findings, including against CONTRIBUTING rules 1–9. Tests were not executed during this read-only review. Residual risk: the PR explicitly marks auto-resume after worker-initiated suspension as unverified. Reviewed commit |
yaythomas
approved these changes
Nov 9, 2026
yaythomas
left a comment
Contributor
There was a problem hiding this comment.
![]()
Compatibility statements and interfaces for earlier preview versions are not necessary while the extras package is alpha.
ParidelPooya
added a commit
to aws/aws-durable-execution-sdk-js
that referenced
this pull request
Nov 9, 2026
…VM (#1182) ## Summary The MicroVM operations no longer resume a suspended MicroVM. The service resumes a MicroVM that was launched with `idlePolicy.autoResumeEnabled: true` when the next request arrives. So the SDK never calls ResumeMicrovm, and the function role no longer needs `lambda:ResumeMicrovm`. This branch is rebased on main after #1178. It keeps the state-check order of #1178: after a sign that the MicroVM does not run, the state check runs first, limited only by the invocation reserve. - `ensureRunning`, its polling loop and its options are removed. - The state check after a sign that the MicroVM does not run is now `checkNotEnded`. It makes one GetMicrovm call. - TERMINATING, TERMINATED and not found fail the job at once with `MicrovmNotRunningError`. - AccessDenied fails it at once with the internal `MicrovmPermissionDeniedError`. Only the GetMicrovm case remains, so its message names `lambda:GetMicrovm`. - RUNNING, PENDING, SUSPENDING and SUSPENDED keep the request step retrying. - Other errors log one warning per step attempt, and the step keeps retrying. - The `suspendCheck` option of `sendJob` is now `stateCheck`. The `recheck` option is removed. - The resume mode is removed: the session launch no longer writes `resumeBeforeDelivery`, the `maybeSuspended` mark after a `suspendAfter` job is gone, and `sendJob` has no `recheck` path. - `suspendAfter`, `autoSuspendOnIdle`, the worker and its hooks are unchanged. The worker README and TSDoc now say that the service resumes the MicroVM. - Docs, TSDoc, the CHANGELOG, the `microvm-session` example and the end-to-end scenarios now set or describe the idle policy. Twin PR: [aws/aws-durable-execution-sdk-python#913](aws/aws-durable-execution-sdk-python#913) ## Design note (CONTRIBUTING rule 3) ### 1. Problem The MicroVM operations resume a suspended MicroVM themselves: GetMicrovm, ResumeMicrovm, polling until RUNNING, a recorded resume mode, and a "may be suspended" mark after `suspendAfter`. This is a large share of the code and of the review rounds, and it needs `lambda:ResumeMicrovm` in the function role. The service can do the same job. ### 2. Cause The SDK was built before `IdlePolicy.autoResumeEnabled` was known and tested. Without it, a suspended MicroVM's endpoint answers `502 MICROVM_CONNECT_FAILED` forever. So the SDK had to resume. ### 3. What a live probe showed (us-east-1, 16 MicroVMs) | Finding | Value | |---|---| | A request to a SUSPENDED MicroVM with `autoResumeEnabled: true` resumes it | Yes. The endpoint holds the request about 750 ms (720 to 1,335 ms, 12 samples), then the worker answers. | | Window right after SuspendMicrovm | For 4.4 to 4.8 s, a request gets `502 MICROVM_CONNECT_FAILED` in about 200 ms and does not resume the MicroVM. GetMicrovm already says SUSPENDED. | | During the suspend hook | `502 INTERNAL_ERROR`, `502 MICROVM_CONNECT_FAILED`, or no answer. A request with no answer reaches the worker after the next resume. The worker dedupes by callback ID. | | `autoResumeEnabled: false`, or no idle policy | `502 MICROVM_CONNECT_FAILED` forever. The MicroVM stays SUSPENDED. | | Idle policy fields | All three are required. `maxIdleDurationSeconds` >= 60, max 2,147,483,647. | | Idle timer | It counts only inbound requests. It suspended a running job after 66 s. | | Worker hooks | The `resume` hook fires on every auto-resume, before the request reaches the worker. The worker's 503 refusal never reached a client after an auto-resume. | The request retries (250 ms doubling to 4 s, 60 s window) reach 7.75 s after 5 retries. So they already cover the 5 s window. With a `retryWindow` under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds. ### 4. Options | Option | Result | |---|---| | A. Keep the SDK resume logic | Works with or without the policy. Keeps the code, the permission and the review cost. | | B. Rely on `autoResumeEnabled`, configured by the developer. **Chosen.** | Removes the resume code and `lambda:ResumeMicrovm`. A developer who suspends must set the policy. | | C. The SDK sets the idle policy itself | Hides a service setting, and must pick `maxIdleDurationSeconds`. A wrong value suspends long jobs. The developer knows their job length. | | D. B plus a check that reads `idlePolicy` from GetMicrovm and fails fast when auto-resume is off | A guard for a developer mistake. Rejected: no guards, and the guideline is documented. | ### 5. Chosen option 1. The SDK never calls ResumeMicrovm. 2. The state check from #1178 stays, only to find a dead MicroVM. After a `502 MICROVM_CONNECT_FAILED` or a request without an answer, it makes one GetMicrovm call, at most once per 2 s. It runs before the retry window and the backoff decide, and only the invocation reserve limits it (#1178). - TERMINATING, TERMINATED, not found: fail at once with `MicrovmNotRunningError`. - RUNNING, PENDING, SUSPENDING, SUSPENDED: keep retrying. The service resumes the MicroVM on a request. - AccessDenied: fail at once with `MicrovmPermissionDeniedError` (#1178). - Other errors: one warning per step attempt, keep retrying. 3. `suspendAfter` (the function calls SuspendMicrovm) and the worker's `autoSuspendOnIdle` stay. Both require the developer's idle policy with `autoResumeEnabled: true`. 4. No check that the policy is set. A developer who suspends without it gets 502 until the retries end: about 6 minutes with the default strategy, then `MicrovmDeliveryError`. 5. Docs (README, TSDoc of `suspendAfter`, `autoSuspendOnIdle`, `idlePolicy`): - To suspend a MicroVM, set `idlePolicy` with `autoResumeEnabled: true`. The SDK does not resume a MicroVM. The service resumes it on the next request. - Set `maxIdleDurationSeconds` above `maximumDurationInSeconds` (the operation timeout plus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job. - Without `autoResumeEnabled`, a job after a suspend fails after its retries with `MicrovmDeliveryError`. - A job that arrives more than about 5 seconds after a suspend takes about 1 s longer. With a `retryWindow` under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds. - Permissions: the function role no longer needs `lambda:ResumeMicrovm`. It needs `lambda:GetMicrovm` for HTTP delivery, and `lambda:SuspendMicrovm` for `suspendAfter`. ### 6. What it removes - `ensureRunning`, its loop `waitUntilRunning`, `EnsureRunningOptions` (`lifecycle.ts`). - The SUSPENDED/SUSPENDING branch and the ResumeMicrovm call in `resumeIfSuspended`, renamed `checkNotEnded`. With it goes the ResumeMicrovm case of `MicrovmPermissionDeniedError`. - The recorded resume mode: `resumeBeforeDelivery` in the launch result and `resumeMode()` (`session.ts`, `shared.ts`). The launch checkpoint field stays readable on replay of old executions, but is no longer written or used. - The `maybeSuspended` mark and the `job.then(mark, mark)` chain after a `suspendAfter` job. - The `recheck` path of `sendJob` (`request.ts`). - `lambda:ResumeMicrovm` from the README permissions, the end-to-end role and the examples guide. Stays: the request retries, the dead-MicroVM check and its order from #1178, `MicrovmPermissionDeniedError`, `suspendAfter`, worker `autoSuspendOnIdle`, the worker's `suspend`/`resume` hooks, its 503 refusal and 30 s grace (still needed for the worker's own SuspendMicrovm), and its callback-ID dedupe (now also needed for late delivery after a frozen request). ### 7. Replay and compatibility - Operation names and subtypes do not change. The event signatures of the `microvm-session` example do not change, so its history file is unchanged. - An execution in flight that launched in resume mode replays its launch checkpoint with `resumeBeforeDelivery: true`. The new code ignores the field. So its next job no longer resumes the MicroVM. That job works only if the MicroVM was launched with `autoResumeEnabled: true`. - An execution in flight whose session used `suspendAfter` without an idle policy recorded `resumeBeforeDelivery: false`. Its next job after the suspend no longer resumes the MicroVM and fails after its retries with `MicrovmDeliveryError`. - The CHANGELOG and the README say both. The package is experimental (`0.1.0-alpha.0`). ## Rebase onto main The branch was first built on the head of #1178 before its last review round. That round changed the same state check: the check runs first after a sign, and the two denial errors became one `MicrovmPermissionDeniedError`. The rebase kept both changes: - `lifecycle.ts`: kept `checkNotEnded` without the resume branch, and kept the one error class `MicrovmPermissionDeniedError`. Its TSDoc no longer mentions ResumeMicrovm. - `request.ts`: kept the check before the window exit and the backoff, budgeted only by the invocation reserve, under the name `stateCheck`. The `recheck` branch is gone. - `shared.ts`: kept the `stateCheck` closure without `maxWaitMs`, `log` and `windowStartedAt`. - Tests: the #1178 tests for the short retry window and the reached reserve are kept, renamed to `stateCheck`. The denied-ResumeMicrovm case of the custom-retry-strategy test is dropped. The composed test that sends a suspend hook before the resend now expects the state check before the first backoff. ## Tests | Package | After | |---|---| | extras (vitest, `tsc --noEmit` first) | 427 tests, 26 files, all pass | | extras coverage | `lifecycle.ts`, `microvm.ts`, `session.ts` 100 %. `request.ts` 100 % lines and branches. All files 97.67 % lines, 96.26 % branches. | | microvm-worker (vitest) | 305, all pass | | examples (vitest) | 276 pass, 3 skipped (cloud only) | | tsc (extras, e2e, examples, worker) | clean | | biome (`npm run lint`) | no findings at error level | New or changed tests: | Test | What it proves | |---|---| | `check-not-ended.test.ts`: RUNNING, PENDING, SUSPENDING, SUSPENDED | One GetMicrovm call, the endpoint is returned, no other call. | | `check-not-ended.test.ts`: TERMINATING, TERMINATED, not found, denied, transient errors, no time left | Fail fast, `MicrovmPermissionDeniedError` naming `lambda:GetMicrovm`, `MicrovmStateCheckError`, the reserve. | | `request-state-check.test.ts` (from #1178) | The check runs when the retry window has no time left, and makes no call at the reserve. | | `microvm-state-check.test.ts`: a delayed 502 or no answer with a 1 s window (from #1178) | `MicrovmNotRunningError` after one GetMicrovm call. | | `microvm-state-check.test.ts`: SUSPENDED or SUSPENDING after a 502 | A `microvm` job keeps sending and succeeds, with no ResumeMicrovm call. | | `session-suspend.test.ts`: MicroVM SUSPENDED or SUSPENDING after the wait | The job gets 502s, one state check, then the fake endpoint auto-resumes it. No ResumeMicrovm call. | | `session-suspend.test.ts`: no state call before a job | With `autoSuspendOnIdle` or an `idlePolicy`, no GetMicrovm call before delivery. | | `session-suspend.test.ts`: launch checkpoint | A new launch records no `resumeBeforeDelivery`. A launch checkpoint of an earlier version with `resumeBeforeDelivery: true` (and `false`) replays, and no job checks the state before delivery. | | `session-suspend-check.test.ts`: the service does not resume | A MicroVM that stays SUSPENDED fails the job with `MicrovmDeliveryError` after the retries, with no ResumeMicrovm call. | | `session-suspend-after.test.ts` | The job after a `suspendAfter` job, also after replay, after a failed job, and when the job was not awaited, is delivered with no ResumeMicrovm call. | | `worker-composed.test.ts` | Against the real worker listener and a fake service that auto-resumes after a 4.8 s settle window: the 503 refusal, the 30 s fallback without the resume hook, a lost answer, and a suspend hook before a resend. | | Fake client | A ResumeMicrovm call records `resume` and throws. So any such call fails the test. | Removed tests: - `lifecycle.test.ts`: every `ensureRunning` test. - `resume-if-suspended.test.ts`: the whole file. Its GetMicrovm cases moved to `check-not-ended.test.ts`. The resume, conflict, poll, wait-budget, ResumeMicrovm-denied, ResumeMicrovm-not-found and other-ResumeMicrovm-error cases are removed. - `microvm-state-check.test.ts`: the resume of a MicroVM suspended from outside, the denied ResumeMicrovm, and the ResumeMicrovm case of the custom-retry-strategy test. - `session-suspend.test.ts` and `session-suspend-check.test.ts`: the resume, conflict, recheck, resume-mode and deploy tests. - `remaining-time.test.ts`: the recheck and shared-window tests. - `retry-strategy-message.test.ts`: the tests for a throttled GetMicrovm in a session. A GetMicrovm error no longer reaches the step retry strategy, so that case no longer exists. - `invocation-reserve.test.ts`: the stalled ResumeMicrovm at the reserve. ## Mutation tests Each mutation was applied alone, and the extras suite was run. | Mutation | Result | |---|---| | M1 `checkNotEnded` calls ResumeMicrovm for SUSPENDED or SUSPENDING | Killed, 20 tests fail | | M2 SUSPENDED fails fast like TERMINATED | Killed, 16 tests fail | | M3 TERMINATED no longer fails fast | Killed, 8 tests fail | | M4 The state check runs after the window exit | Killed, 5 tests fail | | M5 The retry window limits the check's budget | Killed, 2 tests fail | | M6 A denied GetMicrovm is treated as transient | Killed, 7 tests fail | | M7 No 2-second limit between state checks | Killed, 7 tests fail | | M8 No state check after a 502 with `MICROVM_CONNECT_FAILED` | Killed, 42 tests fail | | M9 The endpoint that the state check reports is ignored | Killed, 2 tests fail | | M10 A transient GetMicrovm error fails the step instead of a warning | Killed, 3 tests fail | | M11 A launch checkpoint with `resumeBeforeDelivery: true` makes a state check before delivery | Killed, 1 test fails | | M12 The launch writes `resumeBeforeDelivery` again | Killed, 4 tests fail | ## Live check (us-east-1, dev account, on the rebased branch) The end-to-end function role had `lambda:RunMicrovm`, `lambda:TerminateMicrovm`, `lambda:GetMicrovm`, `lambda:CreateMicrovmAuthToken` and `lambda:SuspendMicrovm`, and no `lambda:ResumeMicrovm`. So any ResumeMicrovm call would have failed the job. | Scenario | Result | |---|---| | `suspend-after-session`: a `suspendAfter` job, then the next job sent at once, idle policy with `autoResumeEnabled` | SUCCEEDED in 15 s. The second job read the value that the first job kept in memory, in the same process. The resume hook fired once. The first `/recall` request got no answer within the 10 s request timeout: it arrived during the suspend hook. The next request resumed the MicroVM and got 202 after 547 ms. | | `suspend-session`: `autoSuspendOnIdle` with a 10 s idle time, idle policy with `autoResumeEnabled` | SUCCEEDED in 90 s. States PENDING, RUNNING, SUSPENDED. The value in memory survived. The resume hook fired once. | Every MicroVM ended TERMINATED. ListMicrovms showed none in another state afterwards. ## Unverified assumptions - In every live run of `suspend-after-session`, the first request after the suspend got no answer, not the 502 of the settle window. So the 502-window retry path ran only in unit tests and in the earlier probe. - "About 6 minutes" until `MicrovmDeliveryError` without `autoResumeEnabled` is computed from the default strategy (5 attempts, each up to a 60 s window, plus 2, 4, 8 and 16 s delays). It was not measured. - The behavior of `suspendedDurationSeconds` when it expires was not tested. The docs ask for the session timeout or more. - Whether outbound traffic, such as heartbeats, resets the idle timer was not tested. The docs ask for a `maxIdleDurationSeconds` above the maximum duration, so it does not matter. ## Decisions not covered by the design note 1. The `suspendCheck` option of `sendJob` is renamed `stateCheck`. 2. `SendJobOptions.windowStartedAt` is removed. Only the state check before delivery used it. 3. The `assertTimeToDeliver` call at the start of the request step is removed. It existed because GetMicrovm ran before `sendJob`, and `sendJob` makes the same check first. 4. The warning after a failed GetMicrovm now says "could not check whether the MicroVM has ended", not "is suspended". 5. A denied GetMicrovm in a session now always records the step error type `MicrovmPermissionDeniedError`. Before, the resume-mode path rethrew the `AccessDeniedException`. 6. Two unreleased CHANGELOG entries of #1178 said that `microvm` uses `lambda:ResumeMicrovm`. They are edited. 7. The end-to-end function role gains `lambda:SuspendMicrovm`, which `suspendAfter` needs and the role lacked. 8. A new default end-to-end scenario, `suspend-after-session`. The logging fetch of the end-to-end function now logs the request path. 9. The `microvm-session` example uses `maxIdleDurationSeconds: 3600` (above its 900 s maximum duration) and `suspendedDurationSeconds: 600` (its timeout). 10. `e2e/probe-lifecycle.mjs` still calls ResumeMicrovm. It probes the service with the operator's credentials, not through the SDK. 11. The `LaunchResult` type drops `resumeBeforeDelivery`. A comment says that an old checkpoint can carry it. 12. The worker README and TSDoc say that the service resumes the MicroVM, and that a 503 makes the durable function send the job again. 13. The reserve error of the state check changes its message. Old: `MicroVM <id> did not reach RUNNING in <n> ms. Last state: unknown.` New, the same in both SDKs: `GetMicrovm for MicroVM <id> did not end before the invocation reserve, after <n> ms.` The message can become the message of the final `MicrovmDeliveryError`, so the checkpoint stores it. 14. `StateCheckOptions` is no longer exported. Only `lifecycle.ts` uses it. ## Follow-up The CI execution role of the integration tests keeps `lambda:ResumeMicrovm` until both SDKs ship this change. A later change in the CI infrastructure removes it.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The service resumes a suspended MicroVM on the next request when its idle policy sets
autoResumeEnabled. So the MicroVM operations no longer resume a MicroVM themselves. The SDK never callsResumeMicrovm, and the function's role no longer needslambda:ResumeMicrovm.This change builds on #910, now on
main. It keeps the order of the state check from #910: after a sign that the MicroVM does not run, the state check runs first, and only the invocation reserve limits it. It keeps the one internal denial error,MicrovmPermissionDeniedError. Only theGetMicrovmcase remains, so its message nameslambda:GetMicrovm.Twin PR: aws/aws-durable-execution-sdk-js#1182
Design note (CONTRIBUTING rule 3)
1. Problem
The MicroVM operations resume a suspended MicroVM themselves:
GetMicrovm,ResumeMicrovm, polling untilRUNNING, a recorded resume mode, and a "may be suspended" mark aftersuspend_after. This is a large share of the code and of the review rounds, and it needslambda:ResumeMicrovmin the function role. The service can do the same job.2. Cause
The SDK was built before
IdlePolicy.autoResumeEnabledwas known and tested. Without it, a suspended MicroVM's endpoint answers502 MICROVM_CONNECT_FAILEDforever, so the SDK had to resume.3. What a live probe showed
A probe ran 16 MicroVMs in us-east-1. Every MicroVM was terminated afterwards.
autoResumeEnabled: trueresumes it502 MICROVM_CONNECT_FAILEDin about 200 ms and does not resume the MicroVM. GetMicrovm already says SUSPENDED.502 INTERNAL_ERROR,502 MICROVM_CONNECT_FAILED, or no answer. A request with no answer reaches the worker after the next resume; the worker dedupes by callback ID.autoResumeEnabled: false, or no idle policy502 MICROVM_CONNECT_FAILEDforever; stays SUSPENDEDmaxIdleDurationSeconds>= 60, max 2,147,483,647.resumehook fires on every auto-resume, before the request reaches the worker. The worker's 503 refusal never reached a client after an auto-resume.The current request retries (250 ms doubling to 4 s, 60 s window) reach 7.75 s after 5 retries, so they already cover the 5 s window. With a
retry_windowunder about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds.4. Options
autoResumeEnabled, configured by the developer. Chosen.lambda:ResumeMicrovm. A developer who suspends must set the policy.maxIdleDurationSeconds; a wrong value suspends long jobs. The developer knows their job length.idlePolicyfrom GetMicrovm and fails fast when auto-resume is off5. Chosen option
502 MICROVM_CONNECT_FAILEDor a request without an answer, the request step makes one GetMicrovm call, at most once per 2 s. The call runs first, before the retry window and the backoff, and only the invocation reserve limits it. With less than 1 s left before the reserve, no call starts, and the attempt ends with the retryableMicrovmEndpointUnavailableError.MicrovmNotRunningError.MicrovmPermissionDeniedError(fix(extras): fail microvm() fast on a dead MicroVM #910 behavior). Its message nameslambda:GetMicrovm.suspend_after(the function calls SuspendMicrovm) and the worker'sauto_suspend_on_idlestay. Both require the developer's idle policy withautoResumeEnabled: true.MicrovmDeliveryError.suspend_after,auto_suspend_on_idle,idle_policy):idle_policywith"autoResumeEnabled": True. The SDK does not resume a MicroVM; the service resumes it on the next request."maxIdleDurationSecondsabovemaximumDurationInSeconds(the sessiontimeoutplus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job."suspendedDurationSecondsto the sessiontimeoutor more." What happens to a job when that time ends is not verified.autoResumeEnabled, a job after a suspend fails after its retries withMicrovmDeliveryError."retry_windowunder about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds."lambda:ResumeMicrovm. It needslambda:GetMicrovmfor HTTP delivery, andlambda:SuspendMicrovmforsuspend_after.6. What it removes
ensure_running, its loop_wait_until_running,EnsureRunningOptions(lifecycle.py).resume_if_suspended. It is renamedcheck_not_ended, and it takes the client, the MicroVM ID, and the remaining-time function.ResumeMicrovmcase ofMicrovmPermissionDeniedError.min(max_wait_seconds, retry_window_seconds)). The check no longer waits, so no wait remains to cap.resume_before_deliveryinLaunchResultandresume_from_config(operation.py,session.py).LaunchResult.from_dictignores aresumeBeforeDeliverykey, so an old checkpoint still replays._may_be_suspendedmark and_resume_modeofMicrovmSession.recheckpath ofsend_job(request.py), and theresumeargument ofdeliver.SendJobOptions.suspend_checkis renamedstate_check, and it takes no argument.SendJobOptions.window_started_at. With no check before delivery, it equaled the start ofsend_job.ResumeMicrovmfromMicrovmApiOperationand frommicrovm_not_found.lambda:ResumeMicrovmfrom the README permissions and the example docs.Stays: the request retries, the dead-MicroVM check and its order,
MicrovmPermissionDeniedError,suspend_after, workerauto_suspend_on_idle, the worker'ssuspend/resumehooks, its 503 refusal and 30 s grace (still needed for the worker's own SuspendMicrovm), and its callback-ID dedupe (now also needed for late delivery after a frozen request).7. Replay and compatibility
resumeBeforeDelivery: true. The new code ignores the field. Its next job no longer resumes the MicroVM. It works only if that MicroVM was launched withautoResumeEnabled: true. Asuspend_aftersession without an idle policy is affected too: its next job after the suspend no longer resumes the MicroVM and fails after its retries withMicrovmDeliveryError. The README says both. The package is unpublished.Source lines (
aws-durable-execution-sdk-python-extras/srconly)Coverage counts 967 statements and 188 branches, down from 1,094 and 228 on
main.Tests
main: 490 passed, 100%)hatch fmt --check.pyfileNew and changed tests:
conftest.pyfakeresume_microvmraises, and the autouse fixtureno_resume_microvmfails any test whose fake client recorded the calllifecycle_test.py::test_check_returns_the_endpoint_of_a_microvm_that_has_not_ended[RUNNING, PENDING, SUSPENDING, SUSPENDED]lifecycle_test.py::test_check_raises_the_state_error_for_a_microvm_that_ends[TERMINATING, TERMINATED],..._for_a_removed_microvmlifecycle_test.py::test_check_fails_at_once_when_get_microvm_is_deniedMicrovmPermissionDeniedError, message nameslambda:GetMicrovmoperation_test.py::test_checks_the_state_after_a_sign_that_uses_up_the_retry_window[connect-failed, no-answer](from #910)operation_test.py::test_ends_an_attempt_of_a_suspended_microvm_without_a_deadlineoperation_test.py::test_a_custom_strategy_sees_the_denial_error_typeMicrovmPermissionDeniedErrorfor a denied GetMicrovmrequest_test.py::test_runs_the_state_check_when_the_sign_used_up_the_windowrequest_test.py::test_starts_no_state_check_within_1_second_of_the_reserveoperation_test.py::test_keeps_sending_the_job_to_a_microvm_suspended_from_outside[SUSPENDING, SUSPENDED]microvmkeeps retryingsession_test.py::test_keeps_sending_to_a_microvm_that_the_worker_suspended_and_uses_its_endpoint[...],test_keeps_sending_the_job_after_a_suspend_after_job[...],test_a_job_after_a_replayed_suspend_after_job_keeps_sending,test_runs_both_items_when_both_find_the_microvm_suspendedsession_test.py::test_replays_a_launch_checkpoint_with_the_resume_mode_of_an_older_releaseresumeBeforeDelivery: truereplays, and the next job worksoperation_test.py::test_launch_result_ignores_the_resume_mode_of_an_older_releasefrom_dictignores the field,to_dictdoes not write itsession_test.py::test_writes_no_resume_mode_in_the_launch_checkpoint[...]worker_composed_test.py(3 tests)resumehook. Jobs succeed without ResumeMicrovm, with and without theresumehook, and when a resend meets a suspend.Removed tests. The behaviors they covered no longer exist:
ensure_runningtest, and the resume, resume-conflict and resume-wait tests of the state check (lifecycle_test.py).lifecycle_test.py::test_check_fails_at_once_when_resume_microvm_is_denied,lifecycle_test.py::test_check_names_no_request_id_for_an_error_of_another_call(the ResumeMicrovm case; the GetMicrovm case stays),operation_test.py::test_the_wait_for_a_resume_gets_a_new_retry_window_without_a_deadline(replaced, see above), and theresumecase ofoperation_test.py::test_a_custom_strategy_sees_one_error_type_for_a_denial.request_test.py::test_counts_the_window_from_the_given_start, withwindow_started_at.rechecktests ofsend_job, the "flag" tests ofsuspend_after, and the ResumeMicrovm scenarios ofrequest_ids_test.py.Edited expectations (rule 5.4):
502 MICROVM_CONNECT_FAILEDfirst, because the check runs only after that sign.request_ids_test.py: "GetMicrovm denies access" expectsMicrovmPermissionDeniedErrorwith its message, because the strict pre-check that raised the rawAccessDeniedExceptionis gone. The message-based retry strategy case for GetMicrovm is removed, because a throttled state check logs a warning and never reaches the strategy.request_test.py::test_checks_the_state_after_a_connect_failure_or_no_answer: the check takes no argument, so the test records when it runs (0 and 3.75 s, before the delay) instead of the time it received.SuspendMicrovmwarning no longer says "The next job checks its state."Mutation checks (rule 4)
Each row changes one branch in
src, runs the extras tests, and restores the file.MICROVM_CONNECT_FAILEDfrom_dictrejects aresumeBeforeDeliverykeyto_dictwritesresumeBeforeDeliveryagainAssumptions (rule 6)
MicrovmDeliveryErrorwithoutautoResumeEnabled: computed from the default retry strategy, not measured.Decisions not covered
maxIdleDurationSecondsis above the maximum duration.suspendedDurationSeconds: 0.auto_suspend_on_idleat all. The service's idle timer with a long job is unsafe, and the worker's own suspend needs the same policy. It stays for now.lambda:ResumeMicrovmgrant in the role that the cloud tests use is defined outside this repository. Remove it in a later change, after both SDKs ship.worker.py, the docstring ofSUSPEND_GRACE_SECONDSinself_suspend.py, two test comments, and the worker README now say that the durable function delivers the job again and the service resumes the MicroVM. The worker code is unchanged.SendJobOptions.suspend_checkis renamedstate_check, andresume_if_suspendedis renamedcheck_not_endedwith a new signature (client, MicroVM ID, remaining-time function).GetMicrovm for MicroVM <id> did not end before the invocation reserve, after <n> ms.The JS twin uses the same message.