Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -837,7 +837,7 @@
},
{
"name": "MicroVM Session Suspend After",
"description": "Runs two jobs in one Lambda MicroVM that its worker suspends after the first job",
"description": "Runs two jobs in one Lambda MicroVM that the function suspends after the first job",
"handler": "microvm_session_suspend_after.handler",
"integration": true,
"usesMicrovm": true,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
"""Run two jobs in one Lambda MicroVM that the worker suspends after the first.
"""Run two jobs in one Lambda MicroVM that the function suspends after the first.

The MicroVM runs the image that scripts/ensure_microvm_image.py builds from
microvm-image/. The image's routes /write and /read write and read a file in
the MicroVM. The image enables the resume hook. So the worker accepts the
next job as soon as the MicroVM resumes.

The session does not suspend when idle. The first job sets suspend_after, so
the worker suspends the MicroVM after that job. A callback named gate holds
the durable function calls SuspendMicrovm after that job's result arrives.
The function's role needs lambda:SuspendMicrovm on the image. A callback named gate holds
the second job. The caller completes the callback with
SendDurableExecutionCallbackSuccess to start the second job.
"""
Expand Down Expand Up @@ -64,8 +65,7 @@ def handler(event: dict[str, Any], context: DurableContext) -> Any:
key: str = event["key"]

def write_gate_read(vm: MicrovmSession, session_context: DurableContext) -> Any:
# The worker suspends the MicroVM when this job has ended and its
# result is reported.
# The session suspends the MicroVM when this job's result arrives.
written = vm.invoke(
session_context,
"write",
Expand All @@ -83,7 +83,7 @@ def write_gate_read(vm: MicrovmSession, session_context: DurableContext) -> Any:
# 1. The first job has ended before the gate starts.
# 2. The second job starts only after the caller completes the gate.
# 3. So no job runs while the gate is pending.
# 4. So the worker suspends the MicroVM after the first job.
# 4. So the session suspends the MicroVM after the first job.
# 5. So a caller that sees the MicroVM SUSPENDED while the gate is
# pending sees a suspend between the two jobs.
# The callback result carries no data. So the handler ignores it.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ def test_my_example(durable_runner):
- Pass a cloud-test input as an object, such as a `dict`. The cloud runner encodes the input with `json.dumps`. So a string that already holds JSON is encoded again, and the handler receives a `str` instead of a `dict`.
- Start a MicroVM test with `run_async`, and read its result with `wait_for_result`. The synchronous `run` uses botocore's 60-second read timeout, and botocore sends the Invoke again after a read timeout. Each resend starts a new execution, which launches its own MicroVM.
- `test_microvm_session_suspend.py` checks that an idle session MicroVM suspends and resumes. The function's role needs `lambda:GetMicrovm` and `lambda:ResumeMicrovm`, and the MicroVM execution role needs `lambda:SuspendMicrovm` on the image.
- `test_microvm_session_suspend_after.py` checks that the worker suspends a session MicroVM after a job with `suspend_after=True`, in a session without auto-suspend, and that the next job resumes it. It needs the same permissions as `test_microvm_session_suspend.py`.
- `test_microvm_session_suspend_after.py` checks that the durable function suspends a session MicroVM after a job with `suspend_after=True`, in a session without auto-suspend, and that the next job resumes it. The function's role needs `lambda:GetMicrovm`, `lambda:ResumeMicrovm`, and `lambda:SuspendMicrovm` on the image. The MicroVM execution role needs no suspend permission.

### Cloud-Only Tests
`@pytest.mark.durable_execution(..., cloud_only=True)` marks a test that needs a service the local runner cannot provide, such as a MicroVM. Local mode skips the test with the reason `cloud_only: runs only against the deployed function`. Cloud mode runs it.
Expand Down
35 changes: 21 additions & 14 deletions packages/aws-durable-execution-sdk-python-extras/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ The worker must support `auto_suspend_on_idle`. The JavaScript worker [`@aws/dur

### Suspending after a job

`MicrovmInvokeOptions(suspend_after=True)` asks the worker to suspend the MicroVM after this job. It is an option of one job, not of the session. `microvm` raises `ValidationError` for it, because `microvm` terminates its MicroVM after its one job. A value that is not a `bool` raises `ValidationError` too, before the job's durable operations start.
`MicrovmInvokeOptions(suspend_after=True)` makes the durable function suspend the MicroVM after this job. It is an option of one job, not of the session. `microvm` raises `ValidationError` for it, because `microvm` terminates its MicroVM after its one job. A value that is not a `bool` raises `ValidationError` too, before the job's durable operations start.

```python
build = vm.invoke(
Expand All @@ -299,13 +299,20 @@ vm.invoke(

What happens:

1. The job request carries `"suspendAfter": true`. The job document stays at version 1, and the field is omitted for every other job. The run hook payload never carries it.
2. The worker acknowledges the field with the header `X-Microvm-Suspend-After: accepted` in its 202 answer.
3. The worker suspends the MicroVM when the job has ended and its outcome is reported, and no other job runs. A job that ends while another job runs leaves the request pending. The worker then suspends the MicroVM when the last running job ends.
4. The worker suspends the MicroVM that its `run` hook named. It never uses the MicroVM ID of a job request.
5. Each later job of the session handle checks the MicroVM state before delivery, and resumes a suspended MicroVM, as with `auto_suspend_on_idle`. The `suspend_after` job itself checks the state too, because the record is set before the job starts.
1. The session calls `SuspendMicrovm` with the session's MicroVM ID when the job's callback result arrives, whether the job succeeded or failed. The call runs inside the job's child context, before the child context returns. A wait for the callback does not suspend the MicroVM.
2. It adds no step and no checkpoint. The child context's own completion checkpoint covers it. On replay, a completed child context does not run again, so the session sends no second suspend.
3. A failed `SuspendMicrovm` call logs a warning and does not fail the job. The next job already checks the state and resumes the MicroVM when needed.
4. The job request carries no field for it, and the worker does nothing for it. The worker needs no change, and the MicroVM's role needs no suspend permission.
5. The session records that the MicroVM may be suspended. Each later job of the session handle checks the MicroVM state before delivery, and resumes a suspended MicroVM, as with `auto_suspend_on_idle`. The record is set after the `suspend_after` job's child context returns or raises, so that job does not check the state before its own delivery.

The setting works with or without `auto_suspend_on_idle`. With both, the worker suspends right after the job, and the idle time applies after each resume.
The setting works with or without `auto_suspend_on_idle`. With both, the function suspends right after the job, and the worker's idle time applies after each resume.

The call is not a durable operation, so these rules are yours to follow. The session adds no guard for them:

1. Do not use it on a job that runs next to other jobs of the session (`map`, `parallel`, promise combinators). Those jobs freeze until the next job resumes the MicroVM, and can hit their heartbeat timeout.
2. The suspend runs inside the job's child context, so it follows the child context replay rules. A result larger than the child context checkpoint limit makes the child context run again on replay, and each run suspends again.
3. A retry after the Lambda invocation fails can send the suspend again. The invocation can also end while the call runs, when other operations wait. The call is then sent again. Suspending a suspended MicroVM succeeds.
4. Adding or removing `suspend_after` changes what the session does on replay. Change it only through a new function version.

The record that a job set `suspend_after` is kept in memory only. A replay of a completed job calls `vm.invoke` again, so it sets the record again. A replay of a completed `map` item, `parallel` branch, or child context does not run its function again. So the next job can skip the state check of a suspended MicroVM. For that case, every session job checks the state after a sign of a suspended MicroVM:

Expand All @@ -321,22 +328,22 @@ A session with `auto_suspend_on_idle` or an `idle_policy`, and a handle after a
Use `suspend_after` only on the job before a long wait, such as a durable wait or a callback:

- Not on back-to-back jobs. Each suspend writes a snapshot, and each resume adds about 1 second before the next job.
- Not on `map` items. A request waits until no job runs. So a `map` with `suspend_after` on its items suspends and resumes between items that do not overlap.
- Not on `map` items or other jobs that run at the same time. The suspend freezes the jobs that still run.
- Not on the last job. The session terminates the MicroVM after its handler returns, so the snapshot is wasted.

It needs:

1. The `resume` hook in the image. Without it, the worker refuses the next job with HTTP 503 for up to 30 seconds after the suspend.
2. A `heartbeat_timeout` on the next job of at least its `retry_window` plus 30 seconds. The heartbeat timeout starts before delivery, so it includes the resume and the refusal.
3. `lambda:GetMicrovm` and `lambda:ResumeMicrovm` in the function's role.
4. `lambda:SuspendMicrovm` on the image in the MicroVM's role. That permission lets code in any MicroVM of the image suspend any other. So use one image per trust boundary.
5. A worker that supports the field. An older worker ignores it, and the MicroVM keeps running. Each execution environment then logs one warning per session: `the worker in MicroVM <id> does not support suspendAfter. The MicroVM keeps running after the job.`
3. `lambda:GetMicrovm`, `lambda:ResumeMicrovm`, and `lambda:SuspendMicrovm` on the image in the function's role. The MicroVM's role needs `lambda:SuspendMicrovm` only for `auto_suspend_on_idle`.

The suspend runs after a job timeout or a heartbeat timeout too, so a job that still runs in the MicroVM is frozen, not stopped.

A suspend that the service refuses for now, such as a throttle or expired credentials, is not retried without `auto_suspend_on_idle`. The MicroVM then keeps running until a later suspend. With `auto_suspend_on_idle`, the worker tries again after the idle time, as it does for an idle suspend.
The call ends at the invocation reserve, like the other calls of the session. A failed suspend, or a call that does not end in time, is not retried. The MicroVM may then still be running until the next job, the idle suspend, or the end of the session. The warning names `lambda:SuspendMicrovm` for a missing permission.

### Session permissions

A session needs the permissions of `microvm` with HTTP delivery: `lambda:RunMicrovm`, `lambda:TerminateMicrovm`, `lambda:CreateMicrovmAuthToken`, `lambda:PassNetworkConnector`, and `iam:PassRole`. Suspending adds the permissions listed above. Grant `lambda:GetMicrovm` to every session's function role. Without it, the check after a sign of a suspended MicroVM logs a warning, and the job keeps its retries.
A session needs the permissions of `microvm` with HTTP delivery: `lambda:RunMicrovm`, `lambda:TerminateMicrovm`, `lambda:CreateMicrovmAuthToken`, `lambda:PassNetworkConnector`, and `iam:PassRole`. Suspending adds the permissions listed above: `lambda:SuspendMicrovm` in the function's role for `suspend_after`, and in the MicroVM's role for `auto_suspend_on_idle`. Grant `lambda:GetMicrovm` to every session's function role. Without it, the check after a sign of a suspended MicroVM logs a warning, and the job keeps its retries.

### Subtypes

Expand Down Expand Up @@ -406,7 +413,7 @@ The MicroVM operations handle the network, the service, and the worker lifecycle
- The idle time is at most the session `timeout`.

A value that breaks any one condition raises `ValidationError` when `microvm_session` is called, before any durable operation starts. See [Suspending an idle MicroVM](#suspending-an-idle-microvm).
4. Set `suspend_after` only on the job before a long wait, and give the next job a `heartbeat_timeout` of at least its `retry_window` plus 30 seconds. On back-to-back jobs and `map` items, each suspend and resume adds latency and a snapshot. A shorter heartbeat timeout can fail the next job with `MicrovmTimeoutError` while the MicroVM resumes. Keep `lambda:GetMicrovm` and `lambda:ResumeMicrovm` in the function's role after a deploy that removes `suspend_after`, until every session that ran such a job has ended. See [Suspending after a job](#suspending-after-a-job).
4. Set `suspend_after` only on the job before a long wait. Never set it on a job that runs next to other jobs of the session. Add `lambda:SuspendMicrovm` to the function's role, and change `suspend_after` only through a new function version. Give the next job a `heartbeat_timeout` of at least its `retry_window` plus 30 seconds. On back-to-back jobs, each suspend and resume adds latency and a snapshot. A shorter heartbeat timeout can fail the next job with `MicrovmTimeoutError` while the MicroVM resumes. Keep `lambda:GetMicrovm` and `lambda:ResumeMicrovm` in the function's role after a deploy that removes `suspend_after`, until every session that ran such a job has ended. See [Suspending after a job](#suspending-after-a-job).
5. Change `auto_suspend_on_idle` or `idle_policy` only through a new version, as in [Deploys and versions](#deploys-and-versions). The launch records the resume mode, and a deploy changes it only for sessions that launch after it. Every job of a session that launched with suspending needs `lambda:GetMicrovm` and `lambda:ResumeMicrovm`, also after a deploy turns suspending off. So remove them from the function's role only as [Suspending an idle MicroVM](#suspending-an-idle-microvm) describes.

### Retry strategies
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"ResumeMicrovm",
"CreateMicrovmAuthToken",
"TerminateMicrovm",
"SuspendMicrovm",
]
"""The Lambda MicroVMs API operations that the MicroVM operations call."""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -360,38 +360,49 @@ class MicrovmInvokeOptions(MicrovmRequestConfig):
"""

suspend_after: bool = False
"""Whether the worker suspends the MicroVM after this job. Defaults to
``False``.

The worker suspends the MicroVM when the job has ended and its outcome is
reported, and no other job runs. A job that ends while another job runs
leaves the request pending. The worker then suspends the MicroVM when
the last running job ends. The next job of the session resumes the
MicroVM before it is delivered. The ``suspend_after`` job itself checks
the state too, because the record is set before the job starts. The
setting works with or without
:attr:`MicrovmSessionConfig.auto_suspend_on_idle`.
"""Whether the durable function suspends the MicroVM after this job.
Defaults to ``False``.

The session calls ``SuspendMicrovm`` for the session's MicroVM when the
job's result arrives, whether the job succeeded or failed. The call runs
inside the job's child context, before the child context returns. It is
not a durable operation of its own. The next job of the session checks
the MicroVM state before delivery, and resumes a suspended MicroVM.

Read these rules before you set it:

1. The durable function suspends the MicroVM after this job's result
arrives.
2. Do not use it on a job that runs next to other jobs of the session
(``map``, ``parallel``, promise combinators). Those jobs freeze until
the next job resumes the MicroVM, and can hit their heartbeat
timeout.
3. The suspend runs inside the job's child context, so it follows the
child context replay rules. A result larger than the child context
checkpoint limit makes the child context run again on replay, and
each run suspends again.
4. A retry after the Lambda invocation fails can send the suspend again.
The invocation can also end while the call runs, when other
operations wait. The call is then sent again. Suspending a suspended
MicroVM succeeds.
5. The function's execution role needs ``lambda:SuspendMicrovm`` on the
MicroVM image.
6. Adding or removing ``suspend_after`` changes what the session does on
replay. Change it only through a new function version.

A failed ``SuspendMicrovm`` call logs a warning and does not fail the
job. The MicroVM may then still be running until the next job or the
end of the session. The call ends at the invocation reserve, like the other
calls of the session, and that counts as a failed call.

Set it on the job before a long wait, such as a durable wait or a
callback. Do not set it on back-to-back jobs, on ``map`` items, or on
the last job of the session. Each suspend and resume costs a snapshot
and about 1 second of latency for the next job. The session terminates
the MicroVM after its last job anyway.

It needs more than a session without suspending:

1. The image enables the ``resume`` hook. Without it, the next job waits
up to 30 seconds of HTTP 503 answers.
2. The function's role has ``lambda:GetMicrovm`` and
``lambda:ResumeMicrovm``.
3. The MicroVM's role has ``lambda:SuspendMicrovm`` on the image.
4. The worker supports the setting. An older worker ignores it, and the
MicroVM keeps running. Each execution environment then logs one
warning per session.

A suspend that the service refuses for now is not retried, unless
:attr:`MicrovmSessionConfig.auto_suspend_on_idle` is ``True``. Only
:meth:`MicrovmSession.invoke` accepts it. :func:`microvm` raises
callback. Do not set it on back-to-back jobs or on the last job of the
session. Each suspend and resume costs a snapshot and about 1 second of
latency for the next job. The session terminates the MicroVM after its
last job anyway. The image must enable the ``resume`` hook, and the
function's role needs ``lambda:GetMicrovm`` and ``lambda:ResumeMicrovm``.

Only :meth:`MicrovmSession.invoke` accepts it. :func:`microvm` raises
``ValidationError`` for it. A value that is not a ``bool`` raises
``ValidationError`` too.
"""
Loading
Loading