Skip to content

feat(extras): suspend after a session job from the durable function - #906

Merged
ParidelPooya merged 9 commits into
mainfrom
feat/microvm-suspend-after-from-function
Nov 5, 2026
Merged

ParidelPooya merged 9 commits into
mainfrom
feat/microvm-suspend-after-from-function

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available

None.

Description of changes

vm.invoke(..., suspend_after=True) now suspends the MicroVM from the durable function, not from the worker in the MicroVM.

  • The session calls SuspendMicrovm with the session's MicroVM ID inside the job's child context. It runs after the job's callback result arrives, on success and on job failure. A wait for the callback (SuspendExecution) does not run it.
  • The call runs through call_before_deadline with the invocation reserve, like GetMicrovm and ResumeMicrovm. A call that does not end by the reserve is a failed call.
  • The call sends no attempt after the wait gave up at the reserve. A before-send handler on the client stops such an attempt, as the worker's self_suspend.py does. Without it, the call's thread could resume in a later invocation, after the next job was delivered, and send or retry SuspendMicrovm with that job inside. Each client gets one handler, registered under a module-level lock before the first suspend on that client, and the client keeps it. Each call keeps its attempt count and its abandoned flag in a threading.local state on the thread that runs the call, and clears it when the call ends. The handler does nothing on a thread without that state. So it leaves other calls of a shared client alone, and an abandoned call does not stop a later call. See "Redesign of the suspend cutoff (rule 8)" below.
  • The call also makes one attempt. An attempt sent before the reserve can still be in flight at the reserve, and a retry adds one more such attempt. So the limit narrows that window, and does not close it. It is a choice under rule 2.5, and it matches the worker's one-attempt client. An attempt that passed the check before the reserve can still reach the service after it. botocore cannot cancel it, and the docstring states this limit.
  • A failed SuspendMicrovm call logs a warning, with the SuspendMicrovm request ID and a note that the MicroVM may still be running and billed until idle suspend or the session ends. The job still returns.
  • After the job's child context returns or fails with a job error, the session marks the MicroVM as possibly suspended. The next job checks the state and resumes it. The suspend_after job itself makes no state check before its own delivery.
  • The job request no longer carries suspendAfter. The session no longer reads the X-Microvm-Suspend-After answer header.
  • The worker no longer handles suspendAfter: no pending-suspend logic, no 202 header, no trigger field in the worker's idle-suspend warning. Idle auto-suspend (auto_suspend_idle_seconds) and the suspend, resume and terminate hooks are unchanged.
  • Permissions move. A function that uses suspend_after needs lambda:SuspendMicrovm on the image in the function's role. The MicroVM's role needs it only for auto_suspend_on_idle.

Design

Design note (rule 3):

  • Problem. A job with suspend_after asked the worker in the MicroVM to suspend its own MicroVM. So the MicroVM's role needed lambda:SuspendMicrovm.
  • Cause. That permission authorizes on the image. So code in any MicroVM of the image can suspend any other MicroVM of the image. The intent was to keep that permission in the durable function and keep the MicroVMs isolated.
  • Options.
    1. Keep the worker-side suspend and narrow its target. This keeps the permission in the MicroVM role, so the problem stays.
    2. Suspend from the function in a new durable step after the job. This adds an operation and a checkpoint for every suspend_after job.
    3. Suspend from the function inside the job's child context, with no new step. The child context's own completion checkpoint covers it.
  • Chosen: option 3. It is also the simplest: it removes the pending-suspend state machine from the worker, the request field and the answer header.
  • What it removes. The suspendAfter field of the job request, the X-Microvm-Suspend-After header and its check, the once-per-session "worker does not support suspendAfter" warning and its bookkeeping, and the worker's pending-suspend logic.

No guards, by design. The session does not check for parallel jobs or for the result size against the child context checkpoint limit. The option's docstring and the README state these rules, as the core step documents its rules:

  1. The durable function suspends the MicroVM after this job's result arrives.
  2. Do not use it on a job that runs next to other jobs of the session (map, parallel, promise combinators). Those jobs freeze until the next job resumes the MicroVM, and can hit their heartbeat timeout.
  3. The suspend runs inside the job's child context, so it follows the child context replay rules. A result larger than the child context checkpoint limit makes the child context run again on replay, and each run suspends again.
  4. A retry after the Lambda invocation fails can send the suspend again. The invocation can also end while the call runs, when other operations wait. The call is then sent again. Suspending a suspended MicroVM succeeds.
  5. The function's execution role needs lambda:SuspendMicrovm on the MicroVM image.
  6. Adding or removing suspend_after changes what the session does on replay. Change it only through a new function version.

Net change in source lines (src/ only): +275 / -420. Public API: no member added. MicrovmInvokeOptions.suspend_after keeps its name and type. Removed from public modules: SUSPEND_AFTER_HEADER and SUSPEND_AFTER_ACCEPTED (extras request), SUSPEND_AFTER_HEADER (worker), MicrovmJobRequest.suspend_after (worker), JobRequest.suspend_after (extras).

Redesign of the suspend cutoff (rule 8)

A review found a new blocking issue in the per-call handler. So the cutoff went back to the design phase under rule 8. This note was agreed before the new code.

  • Problem. A suspend_after SuspendMicrovm call could still retry, or send its first attempt, after the invocation reserve when two sessions suspend at the same time on the shared default client.

  • Cause. Each suspend added a before-send handler with register_first and removed it with unregister. Both clear botocore's HierarchicalEmitter._lookup_cache. _emit reads, computes and stores the handler list for an event without a lock. So thread A can compute the old list, thread B registers (and clears the cache), and A stores the old list back. B's call then reads a cached list without B's handler, and its one-attempt and reserve cutoff never runs. This needs no misuse: two microvm_session calls in a map (one MicroVM each) run on two threads and share the default client of the region.

  • Options.

    Option Result
    A. One handler per client, registered once under a lock before the first suspend on that client, never removed. Each call keeps its state (attempt count, abandoned flag) in threading.local. No add or remove after setup, so no stale list. Works for the default client and a caller's client. Less code: no unique ID, no unregister, no thread ID check.
    B. A new one-attempt client per suspend, as self_suspend.py does Own emitter, no sharing. A caller's custom client cannot be copied cleanly. Extra client setup per suspend.
    C. A lock around register, call and unregister Serializes suspends. A frozen call holds the lock.
    Simplify: drop the cutoff Restores the original bug: a late retry freezes the next job.
  • Chosen: option A.

    1. A module-level lock and a weakref.WeakSet of each client's meta.events record which clients have the handler. The first suspend on a client registers it under the lock, before it sends.
    2. The handler reads the threading.local state. A thread without a state (any other call, or another thread) makes the handler do nothing.
    3. The suspend call sets the state on the thread that runs the call (the call_before_deadline worker thread), clears it in finally, and expired() sets the abandoned flag on that state object.
    4. Rules: one attempt; no attempt after the wait gave up at the reserve.
  • What it removes. The per-call register_first and unregister, the fresh unique_id, and the threading.get_ident() caller check.

  • Remaining window (unchanged). An attempt that passed the check before the reserve can still reach the service after it.

Replay note

The suspend runs inside the child context function. A completed child context returns its checkpointed result on replay and does not run the function again. So a completed job sends no second suspend. A test asserts this across invocations. A child context that did not complete (for example a result over the checkpoint size limit, or a retry after the invocation failed) runs again and can suspend again. Suspending a suspended MicroVM succeeds.

The in-memory "may be suspended" record is set after the child context returns or fails with a job error, not before the job (same as the JS change now). SuspendExecution while the job waits does not set it. A replay of a completed job calls invoke() again and sets it, and the existing check after a 502 MICROVM_CONNECT_FAILED covers a lost record. After a job timeout or heartbeat timeout the suspend still runs, so a job that still runs is frozen, not stopped.

Assumptions

  • SuspendMicrovm on a suspended MicroVM succeeds. Source: the existing worker and README rely on it. Not re-verified against the service in this PR (unverified).
  • The lambda-microvms client has suspend_microvm(microvmIdentifier=...). The worker already calls it, and mypy with boto3-stubs[lambda-microvms] passes.

Decisions not covered by the design note

  • The warning for a failed suspend names the MicroVM and the missing permission. The session logs it with the child context logger.
  • The call catches Exception, so an SDK error from the client never fails the job. SuspendExecution and other BaseException types pass through.
  • The trigger warning field is removed from the worker, since only idle is left.

Tests that fail without each change

Test Fails if
session_test::test_suspends_the_microvm_after_the_job_only_when_it_is_set the suspend is missing, runs before callback.result() (it would run in the waiting invocation too), or targets another ID
session_test::test_suspends_the_microvm_after_the_job_inside_its_child_context the suspend is missing, or a suspend step is added (the test reads the operation names)
session_test::test_suspends_the_microvm_after_a_failed_job_too the job-failure path does not suspend
session_test::test_logs_a_warning_and_returns_the_result_when_the_suspend_fails a failed call raises, logs nothing, or the next job skips the state check
session_test::test_does_not_suspend_a_job_that_did_not_ask_for_it the suspend runs without the option
session_test::test_does_not_suspend_after_a_failed_job_that_did_not_ask_for_it the job-failure path suspends without the option, or a failed job without the option sets the flag (the second job then makes a GetMicrovm)
session_test::test_does_not_suspend_when_the_wait_of_a_suspend_after_job_raises_an_sdk_error a retryable SDK error of the wait sends a suspend
session_test::test_sends_no_second_suspend_when_a_completed_failed_job_replays a replay of a completed failed job sends a second suspend
session_test::test_suspends_and_fails_with_a_timeout_when_a_suspend_after_job_gets_no_result the job-error path does not suspend after a job timeout, which the README promises
session_test::test_ends_the_suspend_at_the_invocation_reserve_and_returns_the_result the call has no deadline, or an expired deadline fails the job
session_test::test_sends_no_suspend_attempt_after_the_reserve_when_the_next_job_runs[held-before-send] the call has no before-send handler, or the handler does not stop an attempt after the reserve. The old call is held before its first attempt past the reserve, the next job is delivered, and the old call is then released
session_test::test_sends_no_suspend_attempt_after_the_reserve_when_the_next_job_runs[answer-lost] the call has no before-send handler. The first attempt is sent before the reserve and loses its answer after the next job is delivered. botocore then retries it
session_test::test_makes_one_suspend_attempt_before_the_reserve_too the handler allows a retry
session_test::test_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_client the call state is shared, not thread-local (the old call's state stops the next call), the handler allows a retry, or the second suspend registers the handler again. The old call is held past the reserve, and the next suspend_after call loses its answer
session_test::test_stops_only_the_attempts_of_its_own_call_on_a_shared_client the handler acts on a thread without a call state, or the call state is shared
session_test::test_keeps_the_cutoff_of_a_suspend_that_starts_while_another_emits[answer-lost] the second suspend loses its one-attempt limit. One suspend is held inside botocore's _emit after it computed its before-send handler list, a second suspend starts, and the first one then stores its list. On the previous head (per-call register and unregister) the second call made 3 attempts
session_test::test_keeps_the_cutoff_of_a_suspend_that_starts_while_another_emits[held-past-reserve] the second suspend sends an attempt after its wait gave up. Same interleaving. On the previous head the second call sent 3 attempts after the reserve
session_test::test_two_first_suspends_on_a_client_register_the_handler_once the registration has no lock, or each suspend registers the handler
session_test::test_leaves_a_later_call_of_the_client_on_the_same_thread_alone the call does not clear its state when it ends
session_test::test_checks_the_state_before_the_job_after_a_failed_suspend_after_job a failed job does not set the flag
session_test::test_sets_the_flag_again_when_a_completed_suspend_after_job_replays a completed job sends a second suspend on replay (checked by moving the call outside the child context)
worker_composed_test::test_suspends_after_a_job_and_the_next_job_resumes_the_microvm the real worker plus the session do not suspend and resume end to end
job_test::test_the_job_request_has_no_suspend_after_field the request carries suspendAfter

I checked the session tests by mutating session.py and operation.py (no suspend on success, no suspend on failure, suspend before the callback result, suspend without the option, suspend outside the child context, a failing call that raises, a failing call that logs nothing, a wrong ID, no state check for the next job). Each mutation fails at least one test above. For the suspend cutoff I mutated operation.py the same way. No lock: test_two_first_suspends_on_a_client_register_the_handler_once fails. Register on each suspend: that test and test_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_client fail. No check for a thread without a call state: test_stops_only_the_attempts_of_its_own_call_on_a_shared_client and test_leaves_a_later_call_of_the_client_on_the_same_thread_alone fail. A shared state instead of threading.local: the race test [answer-lost], the shared-client test and the abandoned test fail. No abandoned check, or expired() does not set the flag: the held-before-send case, the race test [held-past-reserve] and the abandoned test fail. No attempt limit: test_makes_one_suspend_attempt_before_the_reserve_too, the race test [answer-lost] and the abandoned test fail. No clear of the state when the call ends: test_leaves_a_later_call_of_the_client_on_the_same_thread_alone fails. No meta.events check: 10 session tests with a test double fail. The handler checks only that flag. The wait gives up only after the clock passes the reserve, and before the invocation returns. So a clock check would stop the same attempts, and it is not added.

Test counts and coverage:

Package Before After Coverage
extras 457 passed 464 passed 100% (unchanged)
microvm-worker 524 passed 473 passed worker.py and payload.py 100% (unchanged)

The count drops because the removed behavior had tests (worker pending-suspend, header and run-hook-wait cases, the extras acknowledgement and per-session warning cases).

Edited test expectations (rule 5)

  • job_test::test_the_job_fields_fit_in_the_margin_of_the_input_limit: the fixed fields are 92 bytes now, not 112. The 20 bytes were ,"suspendAfter":true.
  • Worker lifecycle_hooks_test: the warning fields no longer include trigger.
  • session_test::SUSPEND_WARNING_START: the failed-suspend warning says that the MicroVM may still be running, not that it keeps running. A timed-out call may have suspended it.
  • Removed: the worker tests test_ignores_a_suspend_after_field_of_a_job_request, test_answers_a_job_request_with_no_suspend_after_header, test_job_request_ignores_a_suspend_after_field and test_run_hook_payload_ignores_suspend_after. They exercised code that no longer exists. job_test::test_the_job_request_has_no_suspend_after_field stays, and checks that the SDK sends no suspendAfter.
  • Removed, because they exercised code that no longer exists: worker tests for the pending suspend, the header, the run-hook wait and the job trigger; extras tests for SUSPEND_AFTER_HEADER, the acknowledgement, the once-per-session warning and old_worker.
  • session_test: assertions that the per-call handler is removed when the call ends are removed. The handler now stays on the client. test_an_abandoned_suspend_does_not_stop_the_next_suspend_on_the_client now asserts that the client has one handler after two suspends.
  • worker_composed_test: the simulated Lambda now suspends through the session's client. The map test with a suspend on one item is removed, since that use is documented as unsupported.

Parity with the JS change

Same design as the JS change in aws/aws-durable-execution-sdk-js. I wrote this PR from the design note, and then compared it with the JS diff at b012ea78:

  • Same behavior: suspend inside the job's child context (subtype MicrovmSessionJob) after the callback result, no new step, no guards, a warning on failure, the session marks the MicroVM as possibly suspended.
  • Same wire change: no suspendAfter in the job request, no X-Microvm-Suspend-After header.
  • Same docstring points (the 6 rules above) and the same permission split.
  • Python names: suspend_after, auto_suspend_on_idle, microvm_session.
  • Parity points: the trigger field removal in the worker warning (the JS change removes it too), the flag timing (set after the child context, as in JS), and the SuspendExecution note (a BaseException raised by callback.result() while it waits, so the suspend runs only after a real result or a job error).

Differences from the JS change

  • The call is not cancelled after the invocation reserve. boto3 cannot cancel a call in flight. The SDK stops waiting at the reserve, and the daemon thread named microvm-call keeps the call until it returns. JS aborts the call. So a before-send handler stops any attempt after the wait gave up at the reserve, and the call makes one attempt. An attempt that was sent before the reserve can still reach the service after it, also in a later invocation after Lambda thaws the thread.
  • Flag after a non-job error. JS sets the flag on any settle of the job (job.then(mark, mark)), an SDK error included. Python sets it for a result and for a ChildContextError only. This shows only when a handler catches an SDK error and runs another job in the same invocation.
  • Suspend after an SDK error during the wait. JS suspends in finally, so an ExecutionError or InvocationError from the wait also suspends. Python lets those two pass through with no suspend. test_does_not_suspend_when_the_wait_of_a_suspend_after_job_raises_an_sdk_error pins this. A normal job outcome behaves the same in both.
  • A delivery failure also sets the "possibly suspended" flag, as in JS. It reaches invoke() as a ChildContextError. So the next job makes one extra GetMicrovm. No SuspendMicrovm is sent for it.

Verification

hatch run dev-extras:test, dev-extras:cov, dev-extras:typecheck, dev-microvm-worker:test, dev-microvm-worker:cov, dev-microvm-worker:typecheck, and hatch fmt --check on the three touched packages. The only finding is an unused variable in packages/aws-durable-execution-sdk-python-microvm-worker/tests/worker_test.py:1155, which this PR does not touch. The cloud example test test_microvm_session_suspend_after.py was not run (it needs a deployed function).

Checklist

  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: not a bug fix
  • Every edited test expectation is listed in the description, with the reason (rule 5)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

vm.invoke(..., suspend_after=True) now calls SuspendMicrovm itself,
inside the job's child context, after the job's result arrives
(success or failure). A failed call logs a warning and the job
still returns. The session then marks the MicroVM as possibly
suspended, so the next job checks the state and resumes it.

- No new step and no new checkpoint.
- The job request no longer carries suspendAfter, and the session
  no longer reads the X-Microvm-Suspend-After answer header.
- The function's role needs lambda:SuspendMicrovm. The MicroVM's
  role needs it only for auto_suspend_on_idle.
The durable function now suspends the MicroVM itself, so the worker
no longer reads suspendAfter from a job request. Removed: the
pending-suspend logic, the X-Microvm-Suspend-After 202 header, and
the job trigger of the suspend-call warning.

Kept: idle auto-suspend, the suspend, resume and terminate hooks.
An older durable function can still send the field. The worker
ignores it.
SuspendMicrovm runs through call_before_deadline with the invocation
reserve. An expired deadline logs the warning, and the job result is
returned.

The may-be-suspended flag is set after the job's child context returns,
or fails with a job error. So the suspend_after job makes no state check
before its own delivery.

The warning carries the SuspendMicrovm request ID and says the MicroVM
is billed until idle suspend or the session ends.

Adds tests for a failed job without suspend_after, a retryable SDK
error, and the replay of a failed job.
@ParidelPooya
ParidelPooya force-pushed the feat/microvm-suspend-after-from-function branch from 8114fb9 to b23cafd Compare November 3, 2026 22:52
A failed job without suspend_after must not set the flag. The test now
runs a second job and asserts it makes no GetMicrovm.

A suspend_after job that gets no result raises MicrovmTimeoutError and
sends exactly one SuspendMicrovm. This is the README promise.

Also note in session.py that a handler can catch the error that ends
the invocation.
The session stops waiting for SuspendMicrovm at the invocation
reserve, and the call goes on in its thread. Lambda freezes that
thread after the invocation returns. A later invocation can then find
the MicroVM RUNNING and deliver the next job. When the old thread
resumes, a first attempt or a botocore retry would suspend the
MicroVM with that job inside.

So a before-send handler, as in the worker's self_suspend.py, stops
every attempt after the first one and any attempt after the reserve.
It acts only on the call's own thread, so other calls of a shared
client go on. The handler is removed when the call ends.

An attempt that was sent before the reserve can still reach the
service after it. The docstring states this limit.

The failed-suspend warning now says that the MicroVM may still be
running, in the code, the README, and the config docstring.
The package is not released yet, so no older durable function sends
suspendAfter. Remove the README text about such a request, and the
three tests that sent the field to the worker.

The extras test that checks that the SDK sends no suspendAfter field
stays.
The before-send handler stopped an attempt when the wait had given
up or when the clock had passed the reserve. The wait gives up only
after the clock passes the reserve, and before the invocation
returns. So each half alone stopped the same attempts, and a test
could not tell them apart. The handler now checks only the flag that
the wait sets when it gives up. Without a deadline the call runs
inline, so no deadline guard is needed.

A new test holds an old suspend call past the reserve on one shared
client. The next job also has suspend_after, and its call loses its
answer. The next call sends exactly one attempt, and the old handler
does not stop it. The test fails with a fixed handler ID and without
the thread check.

The docstring now gives the reason for the thread check: the default
client is cached per Lambda environment, so an abandoned call's
handler stays on it. It describes the one-attempt limit as narrowing
the window, and drops the claim that the timeouts bound the window.
No code reads the suspendAfter field of a run-hook payload. So the
test that checked that the worker ignores it has nothing to check.
Each SuspendMicrovm call added and removed its own before-send handler
on the client. botocore's emitter stores the handler list of an event
without a lock. So a registration while another thread emitted the
event could leave a stored list without the new handler. Two sessions
of a map share the default client. A suspend could then retry, or send
after the invocation reserve.

Now each client gets one handler, registered under a lock before its
first suspend, and keeps it. Each call keeps its attempt count and its
abandoned flag in a thread-local state. The handler does nothing on a
thread without that state. This removes the per-call register and
unregister, the unique ID, and the thread ID check.

A new test holds one suspend inside botocore's emit and starts a second
suspend. The second one makes one attempt, and no attempt after its
wait gave up.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ParidelPooya
ParidelPooya marked this pull request as ready for review November 5, 2026 18:03
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 5, 2026 18:03 — with GitHub Actions Active
@ParidelPooya
ParidelPooya merged commit 46cb38d into main Nov 5, 2026
40 checks passed
@ParidelPooya
ParidelPooya deleted the feat/microvm-suspend-after-from-function branch November 5, 2026 18:03

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 61218a0c Deployed Nov 5, 2026 by ParidelPooya via notify / Generate notification summary #382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants