Repository navigation
feat(extras): suspend after a session job from the durable function - #906
Merged
Merged
Conversation
vm.invoke(..., suspend_after=True) now calls SuspendMicrovm itself, inside the job's child context, after the job's result arrives (success or failure). A failed call logs a warning and the job still returns. The session then marks the MicroVM as possibly suspended, so the next job checks the state and resumes it. - No new step and no new checkpoint. - The job request no longer carries suspendAfter, and the session no longer reads the X-Microvm-Suspend-After answer header. - The function's role needs lambda:SuspendMicrovm. The MicroVM's role needs it only for auto_suspend_on_idle.
The durable function now suspends the MicroVM itself, so the worker no longer reads suspendAfter from a job request. Removed: the pending-suspend logic, the X-Microvm-Suspend-After 202 header, and the job trigger of the suspend-call warning. Kept: idle auto-suspend, the suspend, resume and terminate hooks. An older durable function can still send the field. The worker ignores it.
SuspendMicrovm runs through call_before_deadline with the invocation reserve. An expired deadline logs the warning, and the job result is returned. The may-be-suspended flag is set after the job's child context returns, or fails with a job error. So the suspend_after job makes no state check before its own delivery. The warning carries the SuspendMicrovm request ID and says the MicroVM is billed until idle suspend or the session ends. Adds tests for a failed job without suspend_after, a retryable SDK error, and the replay of a failed job.
ParidelPooya
force-pushed
the
feat/microvm-suspend-after-from-function
branch
from
November 3, 2026 22:52
8114fb9 to
b23cafd
Compare
A failed job without suspend_after must not set the flag. The test now runs a second job and asserts it makes no GetMicrovm. A suspend_after job that gets no result raises MicrovmTimeoutError and sends exactly one SuspendMicrovm. This is the README promise. Also note in session.py that a handler can catch the error that ends the invocation.
yaythomas
reviewed
Nov 4, 2026
The session stops waiting for SuspendMicrovm at the invocation reserve, and the call goes on in its thread. Lambda freezes that thread after the invocation returns. A later invocation can then find the MicroVM RUNNING and deliver the next job. When the old thread resumes, a first attempt or a botocore retry would suspend the MicroVM with that job inside. So a before-send handler, as in the worker's self_suspend.py, stops every attempt after the first one and any attempt after the reserve. It acts only on the call's own thread, so other calls of a shared client go on. The handler is removed when the call ends. An attempt that was sent before the reserve can still reach the service after it. The docstring states this limit. The failed-suspend warning now says that the MicroVM may still be running, in the code, the README, and the config docstring.
The package is not released yet, so no older durable function sends suspendAfter. Remove the README text about such a request, and the three tests that sent the field to the worker. The extras test that checks that the SDK sends no suspendAfter field stays.
The before-send handler stopped an attempt when the wait had given up or when the clock had passed the reserve. The wait gives up only after the clock passes the reserve, and before the invocation returns. So each half alone stopped the same attempts, and a test could not tell them apart. The handler now checks only the flag that the wait sets when it gives up. Without a deadline the call runs inline, so no deadline guard is needed. A new test holds an old suspend call past the reserve on one shared client. The next job also has suspend_after, and its call loses its answer. The next call sends exactly one attempt, and the old handler does not stop it. The test fails with a fixed handler ID and without the thread check. The docstring now gives the reason for the thread check: the default client is cached per Lambda environment, so an abandoned call's handler stays on it. It describes the one-attempt limit as narrowing the window, and drops the claim that the timeouts bound the window.
No code reads the suspendAfter field of a run-hook payload. So the test that checked that the worker ignores it has nothing to check.
yaythomas
reviewed
Nov 5, 2026
Each SuspendMicrovm call added and removed its own before-send handler on the client. botocore's emitter stores the handler list of an event without a lock. So a registration while another thread emitted the event could leave a stored list without the new handler. Two sessions of a map share the default client. A suspend could then retry, or send after the invocation reserve. Now each client gets one handler, registered under a lock before its first suspend, and keeps it. Each call keeps its attempt count and its abandoned flag in a thread-local state. The handler does nothing on a thread without that state. This removes the per-call register and unregister, the unique ID, and the thread ID check. A new test holds one suspend inside botocore's emit and starts a second suspend. The second one makes one attempt, and no attempt after its wait gave up.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available
None.
Description of changes
vm.invoke(..., suspend_after=True)now suspends the MicroVM from the durable function, not from the worker in the MicroVM.SuspendMicrovmwith the session's MicroVM ID inside the job's child context. It runs after the job's callback result arrives, on success and on job failure. A wait for the callback (SuspendExecution) does not run it.call_before_deadlinewith the invocation reserve, likeGetMicrovmandResumeMicrovm. A call that does not end by the reserve is a failed call.before-sendhandler on the client stops such an attempt, as the worker'sself_suspend.pydoes. Without it, the call's thread could resume in a later invocation, after the next job was delivered, and send or retrySuspendMicrovmwith that job inside. Each client gets one handler, registered under a module-level lock before the first suspend on that client, and the client keeps it. Each call keeps its attempt count and its abandoned flag in athreading.localstate on the thread that runs the call, and clears it when the call ends. The handler does nothing on a thread without that state. So it leaves other calls of a shared client alone, and an abandoned call does not stop a later call. See "Redesign of the suspend cutoff (rule 8)" below.SuspendMicrovmcall logs a warning, with theSuspendMicrovmrequest ID and a note that the MicroVM may still be running and billed until idle suspend or the session ends. The job still returns.suspend_afterjob itself makes no state check before its own delivery.suspendAfter. The session no longer reads theX-Microvm-Suspend-Afteranswer header.suspendAfter: no pending-suspend logic, no 202 header, notriggerfield in the worker's idle-suspend warning. Idle auto-suspend (auto_suspend_idle_seconds) and the suspend, resume and terminate hooks are unchanged.suspend_afterneedslambda:SuspendMicrovmon the image in the function's role. The MicroVM's role needs it only forauto_suspend_on_idle.Design
Design note (rule 3):
suspend_afterasked the worker in the MicroVM to suspend its own MicroVM. So the MicroVM's role neededlambda:SuspendMicrovm.suspend_afterjob.suspendAfterfield of the job request, theX-Microvm-Suspend-Afterheader and its check, the once-per-session "worker does not support suspendAfter" warning and its bookkeeping, and the worker's pending-suspend logic.No guards, by design. The session does not check for parallel jobs or for the result size against the child context checkpoint limit. The option's docstring and the README state these rules, as the core
stepdocuments its rules:map,parallel, promise combinators). Those jobs freeze until the next job resumes the MicroVM, and can hit their heartbeat timeout.lambda:SuspendMicrovmon the MicroVM image.suspend_afterchanges what the session does on replay. Change it only through a new function version.Net change in source lines (
src/only): +275 / -420. Public API: no member added.MicrovmInvokeOptions.suspend_afterkeeps its name and type. Removed from public modules:SUSPEND_AFTER_HEADERandSUSPEND_AFTER_ACCEPTED(extrasrequest),SUSPEND_AFTER_HEADER(worker),MicrovmJobRequest.suspend_after(worker),JobRequest.suspend_after(extras).Redesign of the suspend cutoff (rule 8)
A review found a new blocking issue in the per-call handler. So the cutoff went back to the design phase under rule 8. This note was agreed before the new code.
Problem. A
suspend_afterSuspendMicrovmcall could still retry, or send its first attempt, after the invocation reserve when two sessions suspend at the same time on the shared default client.Cause. Each suspend added a
before-sendhandler withregister_firstand removed it withunregister. Both clear botocore'sHierarchicalEmitter._lookup_cache._emitreads, computes and stores the handler list for an event without a lock. So thread A can compute the old list, thread B registers (and clears the cache), and A stores the old list back. B's call then reads a cached list without B's handler, and its one-attempt and reserve cutoff never runs. This needs no misuse: twomicrovm_sessioncalls in amap(one MicroVM each) run on two threads and share the default client of the region.Options.
threading.local.self_suspend.pydoesChosen: option A.
weakref.WeakSetof each client'smeta.eventsrecord which clients have the handler. The first suspend on a client registers it under the lock, before it sends.threading.localstate. A thread without a state (any other call, or another thread) makes the handler do nothing.call_before_deadlineworker thread), clears it infinally, andexpired()sets the abandoned flag on that state object.What it removes. The per-call
register_firstandunregister, the freshunique_id, and thethreading.get_ident()caller check.Remaining window (unchanged). An attempt that passed the check before the reserve can still reach the service after it.
Replay note
The suspend runs inside the child context function. A completed child context returns its checkpointed result on replay and does not run the function again. So a completed job sends no second suspend. A test asserts this across invocations. A child context that did not complete (for example a result over the checkpoint size limit, or a retry after the invocation failed) runs again and can suspend again. Suspending a suspended MicroVM succeeds.
The in-memory "may be suspended" record is set after the child context returns or fails with a job error, not before the job (same as the JS change now).
SuspendExecutionwhile the job waits does not set it. A replay of a completed job callsinvoke()again and sets it, and the existing check after a 502MICROVM_CONNECT_FAILEDcovers a lost record. After a job timeout or heartbeat timeout the suspend still runs, so a job that still runs is frozen, not stopped.Assumptions
SuspendMicrovmon a suspended MicroVM succeeds. Source: the existing worker and README rely on it. Not re-verified against the service in this PR (unverified).lambda-microvmsclient hassuspend_microvm(microvmIdentifier=...). The worker already calls it, and mypy withboto3-stubs[lambda-microvms]passes.Decisions not covered by the design note
Exception, so an SDK error from the client never fails the job.SuspendExecutionand otherBaseExceptiontypes pass through.triggerwarning field is removed from the worker, since onlyidleis left.Tests that fail without each change
session_test::test_suspends_the_microvm_after_the_job_only_when_it_is_setcallback.result()(it would run in the waiting invocation too), or targets another IDsession_test::test_suspends_the_microvm_after_the_job_inside_its_child_contextsession_test::test_suspends_the_microvm_after_a_failed_job_toosession_test::test_logs_a_warning_and_returns_the_result_when_the_suspend_failssession_test::test_does_not_suspend_a_job_that_did_not_ask_for_itsession_test::test_does_not_suspend_after_a_failed_job_that_did_not_ask_for_itGetMicrovm)session_test::test_does_not_suspend_when_the_wait_of_a_suspend_after_job_raises_an_sdk_errorsession_test::test_sends_no_second_suspend_when_a_completed_failed_job_replayssession_test::test_suspends_and_fails_with_a_timeout_when_a_suspend_after_job_gets_no_resultsession_test::test_ends_the_suspend_at_the_invocation_reserve_and_returns_the_resultsession_test::test_sends_no_suspend_attempt_after_the_reserve_when_the_next_job_runs[held-before-send]before-sendhandler, or the handler does not stop an attempt after the reserve. The old call is held before its first attempt past the reserve, the next job is delivered, and the old call is then releasedsession_test::test_sends_no_suspend_attempt_after_the_reserve_when_the_next_job_runs[answer-lost]before-sendhandler. The first attempt is sent before the reserve and loses its answer after the next job is delivered. botocore then retries itsession_test::test_makes_one_suspend_attempt_before_the_reserve_toosession_test::test_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_clientsuspend_aftercall loses its answersession_test::test_stops_only_the_attempts_of_its_own_call_on_a_shared_clientsession_test::test_keeps_the_cutoff_of_a_suspend_that_starts_while_another_emits[answer-lost]_emitafter it computed itsbefore-sendhandler list, a second suspend starts, and the first one then stores its list. On the previous head (per-call register and unregister) the second call made 3 attemptssession_test::test_keeps_the_cutoff_of_a_suspend_that_starts_while_another_emits[held-past-reserve]session_test::test_two_first_suspends_on_a_client_register_the_handler_oncesession_test::test_leaves_a_later_call_of_the_client_on_the_same_thread_alonesession_test::test_checks_the_state_before_the_job_after_a_failed_suspend_after_jobsession_test::test_sets_the_flag_again_when_a_completed_suspend_after_job_replaysworker_composed_test::test_suspends_after_a_job_and_the_next_job_resumes_the_microvmjob_test::test_the_job_request_has_no_suspend_after_fieldsuspendAfterI checked the session tests by mutating
session.pyandoperation.py(no suspend on success, no suspend on failure, suspend before the callback result, suspend without the option, suspend outside the child context, a failing call that raises, a failing call that logs nothing, a wrong ID, no state check for the next job). Each mutation fails at least one test above. For the suspend cutoff I mutatedoperation.pythe same way. No lock:test_two_first_suspends_on_a_client_register_the_handler_oncefails. Register on each suspend: that test andtest_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_clientfail. No check for a thread without a call state:test_stops_only_the_attempts_of_its_own_call_on_a_shared_clientandtest_leaves_a_later_call_of_the_client_on_the_same_thread_alonefail. A shared state instead ofthreading.local: the race test[answer-lost], the shared-client test and the abandoned test fail. No abandoned check, orexpired()does not set the flag: theheld-before-sendcase, the race test[held-past-reserve]and the abandoned test fail. No attempt limit:test_makes_one_suspend_attempt_before_the_reserve_too, the race test[answer-lost]and the abandoned test fail. No clear of the state when the call ends:test_leaves_a_later_call_of_the_client_on_the_same_thread_alonefails. Nometa.eventscheck: 10 session tests with a test double fail. The handler checks only that flag. The wait gives up only after the clock passes the reserve, and before the invocation returns. So a clock check would stop the same attempts, and it is not added.Test counts and coverage:
worker.pyandpayload.py100% (unchanged)The count drops because the removed behavior had tests (worker pending-suspend, header and run-hook-wait cases, the extras acknowledgement and per-session warning cases).
Edited test expectations (rule 5)
job_test::test_the_job_fields_fit_in_the_margin_of_the_input_limit: the fixed fields are 92 bytes now, not 112. The 20 bytes were,"suspendAfter":true.lifecycle_hooks_test: the warning fields no longer includetrigger.session_test::SUSPEND_WARNING_START: the failed-suspend warning says that the MicroVM may still be running, not that it keeps running. A timed-out call may have suspended it.test_ignores_a_suspend_after_field_of_a_job_request,test_answers_a_job_request_with_no_suspend_after_header,test_job_request_ignores_a_suspend_after_fieldandtest_run_hook_payload_ignores_suspend_after. They exercised code that no longer exists.job_test::test_the_job_request_has_no_suspend_after_fieldstays, and checks that the SDK sends nosuspendAfter.SUSPEND_AFTER_HEADER, the acknowledgement, the once-per-session warning andold_worker.session_test: assertions that the per-call handler is removed when the call ends are removed. The handler now stays on the client.test_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_clientnow asserts that the client has one handler after two suspends.worker_composed_test: the simulated Lambda now suspends through the session's client. The map test with a suspend on one item is removed, since that use is documented as unsupported.Parity with the JS change
Same design as the JS change in
aws/aws-durable-execution-sdk-js. I wrote this PR from the design note, and then compared it with the JS diff atb012ea78:MicrovmSessionJob) after the callback result, no new step, no guards, a warning on failure, the session marks the MicroVM as possibly suspended.suspendAfterin the job request, noX-Microvm-Suspend-Afterheader.suspend_after,auto_suspend_on_idle,microvm_session.triggerfield removal in the worker warning (the JS change removes it too), the flag timing (set after the child context, as in JS), and theSuspendExecutionnote (aBaseExceptionraised bycallback.result()while it waits, so the suspend runs only after a real result or a job error).Differences from the JS change
microvm-callkeeps the call until it returns. JS aborts the call. So abefore-sendhandler stops any attempt after the wait gave up at the reserve, and the call makes one attempt. An attempt that was sent before the reserve can still reach the service after it, also in a later invocation after Lambda thaws the thread.job.then(mark, mark)), an SDK error included. Python sets it for a result and for aChildContextErroronly. This shows only when a handler catches an SDK error and runs another job in the same invocation.finally, so anExecutionErrororInvocationErrorfrom the wait also suspends. Python lets those two pass through with no suspend.test_does_not_suspend_when_the_wait_of_a_suspend_after_job_raises_an_sdk_errorpins this. A normal job outcome behaves the same in both.invoke()as aChildContextError. So the next job makes one extraGetMicrovm. NoSuspendMicrovmis sent for it.Verification
hatch run dev-extras:test,dev-extras:cov,dev-extras:typecheck,dev-microvm-worker:test,dev-microvm-worker:cov,dev-microvm-worker:typecheck, andhatch fmt --checkon the three touched packages. The only finding is an unused variable inpackages/aws-durable-execution-sdk-python-microvm-worker/tests/worker_test.py:1155, which this PR does not touch. The cloud example testtest_microvm_session_suspend_after.pywas not run (it needs a deployed function).Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.