From 03538bdfbeb764a952a99cc8d78f863074449700 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Mon, 9 Nov 2026 03:00:46 +0000 Subject: [PATCH] docs: make package READMEs lean and usage-first Rewrite the core, extras, microvm-worker, and testing READMEs. Each README now has install, a quick start, common uses, short advanced notes, and links to reference docs. Move the long sections to docs files next to each package: - core: docs/plugins.md - extras: docs/microvm.md - microvm-worker: docs/worker.md - testing: callback-failure notes into docs/error-responses.md Point the 2.x to 3.0 migration guide links at docs/plugins.md. Update the extras subtypes note: the core SDK now accepts a custom sub_type. Replace the deprecated run(timeout=...) in the testing example with execution_timeout. --- docs/migration-2.x-to-3.0.md | 4 +- .../README.md | 421 ++++------------ .../docs/microvm.md | 466 ++++++++++++++++++ .../README.md | 260 ++++------ .../docs/worker.md | 196 ++++++++ .../README.md | 190 +++---- .../docs/error-responses.md | 18 +- .../README.md | 366 ++++++-------- .../docs/plugins.md | 213 ++++++++ 9 files changed, 1291 insertions(+), 843 deletions(-) create mode 100644 packages/aws-durable-execution-sdk-python-extras/docs/microvm.md create mode 100644 packages/aws-durable-execution-sdk-python-microvm-worker/docs/worker.md create mode 100644 packages/aws-durable-execution-sdk-python/docs/plugins.md diff --git a/docs/migration-2.x-to-3.0.md b/docs/migration-2.x-to-3.0.md index 3ce5e4b03..24b9dab89 100644 --- a/docs/migration-2.x-to-3.0.md +++ b/docs/migration-2.x-to-3.0.md @@ -194,7 +194,7 @@ Provider names must be unique across installed distributions. Explicit factories run before environment-selected factories, and selecting the same factory object through both paths deduplicates it by identity. Separate factory instances are separate registrations, even if they construct the same plugin class. See -[dynamic instrumentation plugins](../packages/aws-durable-execution-sdk-python/README.md#dynamic-instrumentation-plugins). +[dynamic instrumentation plugins](../packages/aws-durable-execution-sdk-python/docs/plugins.md). ### Migrate OpenTelemetry registrations @@ -345,7 +345,7 @@ operation's end hook returns before the next operation's start hook is called. Threads are not ordered with each other, so guard state shared between hooks with a lock. The `DurableInstrumentationPlugin` docstring states where each hook runs and the exceptions to this order. See the -[invocation worker lifecycle](../packages/aws-durable-execution-sdk-python/README.md#invocation-worker-lifecycle) +[invocation worker lifecycle](../packages/aws-durable-execution-sdk-python/docs/plugins.md#invocation-worker-lifecycle) for context-variable handling. ## Recommended validation after upgrading diff --git a/packages/aws-durable-execution-sdk-python-extras/README.md b/packages/aws-durable-execution-sdk-python-extras/README.md index cf0faa064..feeb5ce02 100644 --- a/packages/aws-durable-execution-sdk-python-extras/README.md +++ b/packages/aws-durable-execution-sdk-python-extras/README.md @@ -2,13 +2,15 @@ > This package is experimental. Its API can change in any release. -Custom operations for the [AWS Durable Execution SDK for Python](../aws-durable-execution-sdk-python). Each operation composes the SDK's built-in operations and calls AWS APIs. The operations live in this package, so the core SDK does not depend on those APIs' clients. +Run jobs from a durable function in AWS Lambda MicroVMs, with the `microvm` and `microvm_session` operations. -The package is the Python counterpart of [`@aws/durable-execution-sdk-js-extras`](https://github.com/aws/aws-durable-execution-sdk-js/tree/main/packages/aws-durable-execution-sdk-js-extras). +## Install -## `microvm`: run a job in an AWS Lambda MicroVM +```console +pip install aws-durable-execution-sdk-python-extras +``` -`microvm` launches a new [Lambda MicroVM](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html), delivers the job to it, waits for the job's result, and terminates the MicroVM. The function invocation ends while the MicroVM works, so no Lambda compute is billed during the wait. +## Quick start ```python import os @@ -31,145 +33,17 @@ def handler(event: dict, context: DurableContext) -> dict: return microvm(context, "build", job_input={"repo": event["repo"]}, config=CONFIG) ``` -The first argument is the context that the call runs in. Inside a child context or a `map` item, pass that context, so the operation's steps belong to it. `job_input` and `config` are keyword-only. A job can be a string, and so is the name. So positional arguments in the wrong order would pass validation and run the wrong job. - -The code in the MicroVM receives the job and reports its result. The worker package [`aws-durable-execution-sdk-python-microvm-worker`](../aws-durable-execution-sdk-python-microvm-worker) does this, and so does the JavaScript worker. The job document is the same in every SDK language. - -### How the job reaches the MicroVM - -The operation picks the delivery from the input size and the config: +`microvm` launches a [Lambda MicroVM](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html), sends it the job, waits for the result, and terminates the MicroVM. The function invocation ends while the MicroVM works. So no Lambda compute is billed during the wait. -1. A `microvm` job goes in the `RunMicrovm` request when its `run` hook payload fits in 4,096 Unicode code points and its config sets no `request.path`. The payload holds the input, the callback ID, and a few other fields. Lambda passes the payload to the worker's `run` lifecycle hook. The MicroVM gets the `NO_INGRESS` connector, so nothing can reach it. -2. Every other job goes over HTTP: - - A `microvm` job whose `run` hook payload is over 4,096 code points. - - A `microvm` job whose config sets `request.path`, whatever its size. - - Every session job. +`job_input` and `config` are keyword-only. Inside a child context or a `map` item, pass that context as the first argument. - The launch request carries no job, and a request step POSTs the job to the MicroVM's endpoint after it starts. The Python worker accepts a request body of at most 6 MiB. The MicroVM gets the `ALL_INGRESS` connector, and every request carries a short-lived auth token that the operation creates. +The image runs the [MicroVM worker](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-microvm-worker). The worker receives the job and reports its result. -The worker package runs its `handler` for both deliveries. So the code in the MicroVM does not change with the input size. +## Common uses -The HTTP delivery waits inside the Lambda invocation until the MicroVM's endpoint accepts the job, usually one to two seconds. The `run` hook delivery adds no Lambda time for the boot. +### Run several jobs in one MicroVM -An application that already serves HTTP routes can receive every job on its own route: - -```python -import dataclasses - -from aws_durable_execution_sdk_python_extras.microvm import MicrovmRequestConfig - -config = dataclasses.replace(CONFIG, request=MicrovmRequestConfig(path="/clone-build")) -microvm(context, "build", job_input=job_input, config=config) -``` - -A `path` sends every job over HTTP, whatever its size. The route must answer with a 2xx status at once and run the job in the background. `port` names a port other than 8080. `retry_window` sets how long the request step retries inside one attempt, 60 seconds by default. - -The same job can arrive again, also after it ended, so ignore a second request for a callback ID that you already ran. The worker package does this. A route served without it must do it itself. A job arrives again in these cases: - -1. A request reaches the route, and its answer is lost. -2. The request step stops waiting for a request after at most 10 seconds, but the request keeps running in its thread and reaches the route. The next attempt sends the job again. -3. The invocation ends after the route accepted the job, and before the request step checkpointed its success. The step then runs again in the next invocation. - -The path must be an RFC 3986 path without a `.` or `..` segment, and the operation sends it exactly as written. A dot segment is rejected also when the dot is written as `%2e`, because the JavaScript SDK's `fetch` resolves such a segment, and a proxy or server can too. So both SDKs accept the same paths. Percent-encode any other character, for example a space as `%20`. The workers match the raw request path, so register the same string as the worker's route. An invalid path raises `ValidationError` before any durable operation starts. - -### What it does - -The operation runs its durable operations in one child context named after the second argument: - -1. `.callback` creates a durable callback. The service generates the callback ID, and the SDK checkpoints it. So the ID is unique to this call and stays the same on replay. -2. `.launch` calls `RunMicrovm`. The client token is the SHA-256 hex digest of the callback ID. So a retried or replayed launch returns the same MicroVM instead of starting a second one. With `run` hook delivery, the launch request contains the job. The step checkpoints plain JSON, `{"microvmId": "...", "endpoint": "...", "delivery": "http"}`, as the JavaScript operation does. The `endpoint` is present only for HTTP delivery. So a reader of the execution history finds the MicroVM ID there. -3. With HTTP delivery, `.request` creates an auth token and POSTs the job. -4. The child context waits for the MicroVM to complete the callback. -5. `.terminate` calls `TerminateMicrovm`. It runs after success, failure, and timeout alike, and after a failed delivery. - -Replay follows the delivery that the launch checkpointed, not a new choice. A deploy between the launch and a replay can change the input or the config. The launch result comes from the checkpoint, so the job still goes the way the MicroVM was launched for. - -A terminate failure does not fail the operation. The job's outcome is already recorded at that point. So the operation logs an error instead of raising. The MicroVM then keeps running, and is billed, until the platform ends it at its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. - -The terminate step runs only when the operation sees the job's outcome. In these cases it never does, and the MicroVM also runs until its maximum duration: - -- A `map` or `parallel` completes early, for example with `min_successful`, while an item still waits for its MicroVM. The SDK never runs that item again. -- The execution fails, times out, or is stopped while the operation waits. -- A launch attempt that timed out or lost its connection can still have created a MicroVM. When the launch step still fails, the operation never learns that MicroVM's ID. So it cannot terminate that MicroVM. - -### Errors - -Every MicroVM failure is a `MicrovmError`. Catch the base class for all of them, or a subclass for one kind: - -| Error | When | -|---|---| -| `MicrovmLaunchError` | `RunMicrovm` failed after all retries, or the launch request was not valid, for example with an empty `image_identifier` or `execution_role_arn`. A MicroVM can still run the job. The list below the table says when. | -| `MicrovmDeliveryError` | HTTP delivery failed after all retries, or the route answered a 4xx status such as 404, or a redirect, or GetMicrovm was denied (the message names the permission). A job after a suspend fails this way after its retries when the session's `idle_policy` does not set `autoResumeEnabled`. | -| `MicrovmNotRunningError` | A `MicrovmDeliveryError`: the MicroVM is terminating, terminated, or removed. It is not retried. Any job over HTTP, from `microvm` or a session, reports it after an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED` or a request without an answer: the request step then calls `GetMicrovm` after each such answer, at most once per 2 seconds. The endpoint of a terminated MicroVM answers that 502, measured in us-east-1. The message names the state, the `stateReason`, and the `GetMicrovm` request ID. Any job over HTTP also reports it when `CreateMicrovmAuthToken` reports the MicroVM removed. A job in the `run` hook has no request step, so a MicroVM that ends before the job runs fails it with `MicrovmTimeoutError`. See [The state check](#the-state-check). | -| `MicrovmJobFailedError` | The job handler in the MicroVM failed, or the MicroVM sent a result that is not valid JSON. The message includes the error type that the MicroVM reported. | -| `MicrovmTimeoutError` | No result arrived within `timeout`, or no heartbeat within `heartbeat_timeout`. A MicroVM that crashed ends this way. | - -After `MicrovmLaunchError`, the delivery decides whether a MicroVM runs the job: - -1. With `run` hook delivery, the launch request contains the job. A launch attempt that timed out or lost its connection can still have created a MicroVM. That MicroVM then runs the job. So running the job again after this error can run it twice. -2. With HTTP delivery, the launch request contains no job. The request step starts only after a successful launch. So when you invoke through a version or an alias, no MicroVM runs the job after an HTTP launch fails. - -```python -from aws_durable_execution_sdk_python_extras.microvm import ( - MicrovmError, - MicrovmTimeoutError, -) - -try: - result = microvm(context, "build", job_input=job_input, config=CONFIG) -except MicrovmTimeoutError: - ... # The MicroVM crashed, or took too long. -except MicrovmError: - ... # Any other MicroVM failure. -``` - -The class is the same on the first run and on every replay, so code that branches on it stays deterministic. A `microvm` call inside your own child context, or inside a `map` or `parallel` item, reaches the code around that context as the SDK's `ChildContextError`. Its `__cause__` is the MicroVM error, so check `isinstance(error.__cause__, MicrovmTimeoutError)` there. The operation raises three errors before any durable operation starts. An invalid name or config raises the SDK's `ValidationError`. An input that is not JSON-serializable raises `TypeError`. An input over `MAX_JOB_INPUT_BYTES` raises `ValueError`. `Duration` raises the same `ValidationError` for a negative value. The operation does not check `image_identifier` and `execution_role_arn` before it starts. boto3 checks them against the API model before it sends `RunMicrovm`, and the service checks them again. So an invalid value fails the launch step at once with `MicrovmLaunchError`, after the callback step. - -A failed AWS call keeps its request ID in the error message, in the form ` request ID: .`, for example `RunMicrovm request ID: .`. A support case can quote it. This applies to `RunMicrovm`, `GetMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The SDK checkpoints the message of a failed step, and not its `__cause__`. So the request ID is in the message, the handler sees it on the first run and on replay, and the execution history keeps it. A failed `TerminateMicrovm` call does not fail the operation. So a `TerminateMicrovm` request ID appears only in the terminate log entry and in the terminate step record. A message has no request ID when the answer carries none, for example after a connection failure. A message also has no request ID when the error is not the answer of the Lambda MicroVMs call. For example, a `client` with role-assuming credentials calls STS `AssumeRole` inside each call, and an STS error carries the request ID of STS. The operation recognizes such an error by its `operation_name`. The step raises a copy of the botocore error, of the same class, with the same `response`, and with the request ID in its message. So `default_microvm_retry_strategy` decides as before. A custom `retry_strategy` receives the error that botocore raised, without the request ID. So a strategy that matches the message, such as `create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))`, decides on the message of the service. A `ResourceNotFoundException` from `GetMicrovm` or `CreateMicrovmAuthToken` is the exception. It means that the MicroVM no longer exists. So the operation raises `MicrovmStateError` instead, and the strategy receives that error. Its `__cause__` is the botocore error. An error that the operation creates itself, such as `MicrovmStateError` or "RunMicrovm returned no microvmId", has the request ID in its message when the strategy receives it. A custom client can raise one error object for several calls. Each step message then names only its own call, and the object keeps its message. - -When the error escapes the handler, the execution's error type is the class's fully-qualified name, such as `aws_durable_execution_sdk_python_extras.microvm.errors.MicrovmJobFailedError`. The SDK uses that name to rebuild the class on replay. The JavaScript operation reports the short name, `MicrovmJobFailedError`. So a rule that matches the error type of both SDKs, such as a Step Functions `Catch` or a metric filter, must match both forms. For example, match on the suffix. - -### Retries in two tiers - -A step retry costs one durable operation and one more Lambda invocation. Most failures here are short, so the steps retry in two tiers: - -1. **Inside one step attempt.** The boto3 client retries throttling and server errors of `RunMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The request step retries the POST itself: connection errors, timeouts, 429, and 5xx, with a delay from 250 milliseconds to 4 seconds. It retries one 401 or 403 with a new auth token. The endpoint answers this way until the MicroVM's `run` hook returns. This tier ends after `retry_window`, or 10 seconds before the invocation times out, whichever comes first. -2. **The step retry strategy.** When the first tier gives up, the step retries after a longer delay, outside the invocation. `default_microvm_retry_strategy` is the default, unless the config sets `retry_strategy`. It retries throttling, server errors, connection failures, timeouts, and the end of the first tier, for at most 5 attempts, with a delay of 2, 4, 8, and 16 seconds. Every other error, such as `ValidationException`, `AccessDeniedException`, or `InsufficientCapacityException`, fails the step at once. - -A route that answers another 4xx status fails the operation without a retry, with `MicrovmDeliveryError`. So does a redirect, 3xx. The operation does not follow redirects. The execution history stores the error message. A query in the `Location` can hold a secret. So the message names the path, and the status of a 3xx answer. It omits the `Location`, and the URL that a followed redirect reached. Set the path that the route serves, for example with its trailing slash. The JavaScript operation also fails every 3xx at once, with `MicrovmDeliveryError`. - -The POST ignores the proxy settings in the environment, as the JavaScript SDK's `fetch` does. boto3 reads `HTTPS_PROXY`. So in a network that reaches the MicroVM endpoint only through a proxy, the launch succeeds and every POST fails. Pass an `opener` in the config for that network, for example `urllib.request.build_opener(urllib.request.ProxyHandler({"https": "http://proxy.internal:3128"}))`. When an opener follows a redirect, the delivery still fails at once with `MicrovmDeliveryError`. The followed request carries no auth token, because the operation adds its headers as unredirected headers. - -The 10-second reserve keeps time for the step to record its failure. A Lambda timeout during a step attempt records no outcome for the attempt. So every call of the first tier must end before the reserve. boto3 cannot cancel a call, so a `CreateMicrovmAuthToken` call runs in a separate thread, and the step stops waiting for it at the reserve. Each POST runs in a separate thread too, and the step waits for it at most 10 seconds, or until the reserve. A socket timeout alone would not bound it, because each byte received starts the socket timeout again. - -### Defaults and limits - -- `timeout` must be between 1 second and 8 hours, because a MicroVM lives at most 8 hours. The MicroVM's `maximumDurationInSeconds` is `timeout` plus 5 minutes, at most 8 hours. So a timeout above 7 hours 55 minutes leaves the MicroVM less than 5 minutes beyond the callback's timeout. -- `job_input` is sent as JSON. So the MicroVM receives what `json.dumps` writes: a tuple arrives as a list, and a dict key that is not a string arrives as a string. -- The worker accepts a job request body of at most 6 MiB. The body holds the JSON of `job_input` and a few job fields. The job fields take less than 3 KiB. So `job_input` can be at most `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, as UTF-8 JSON. A larger input raises `ValueError` from `microvm` or `vm.invoke` before any durable operation starts. -- The MicroVM gets the `INTERNET_EGRESS` connector, because it needs egress to reach the Lambda callback APIs. For ingress, it gets `NO_INGRESS` with `run` hook delivery and `ALL_INGRESS` with HTTP delivery. If you set `ingress_network_connectors`, allow ingress for every job that goes over HTTP. A `microvm` job goes over HTTP when its `run` hook payload is over 4,096 code points, or when its config sets `request.path`. Every session job goes over HTTP. An empty tuple sends no connector. -- `heartbeat_timeout` starts when the callback is created, before the launch. The worker sends its first heartbeat when the job arrives. So the first period must cover the launch, the boot, and, for HTTP delivery, the request step, each with its retries. A throttled launch can take several minutes, and the request step retries for up to `retry_window` in each attempt. With a shorter heartbeat timeout, the operation fails with `MicrovmTimeoutError` before the MicroVM sends its first heartbeat. -- The request step needs at least 11 seconds left in the invocation when it starts. It keeps a 10-second reserve before the invocation times out, and it starts a call only with at least 1 second left before the reserve. The launch step can run earlier in the same invocation, so the function timeout must cover it too. A function timeout of `retry_window` plus 11 seconds, 71 seconds by default, leaves the first retry tier about its whole window. With less than 11 seconds left, every attempt fails with `MicrovmEndpointUnavailableError`, and the message of the final `MicrovmDeliveryError` names the function timeout. The Lambda default timeout is 3 seconds. With it, `microvm` runs small jobs in the `run` hook and fails every job that goes over HTTP: a job too large for the hook, and a job with `request.path`. -- The Region comes from the durable execution ARN, or from `AWS_REGION`. -- Without a `client`, the operation creates one `lambda-microvms` client per Region, on first use. It has a 5-second connect timeout, a 30-second read timeout, and 3 attempts in botocore's standard retry mode. So one launch or terminate attempt takes at most about 2 minutes. A call that runs past the function's timeout records no outcome for its step attempt, and the step runs again. The launch is idempotent through its client token, so it is repeated, not doubled. - -### Permissions - -The function's execution role needs: - -- `lambda:RunMicrovm` and `lambda:TerminateMicrovm` on the MicroVM image. Without `lambda:TerminateMicrovm`, every operation still returns its result, but every MicroVM runs until its timeout plus 5 minutes, at most 8 hours. The only sign is an error in the function's log. -- `lambda:CreateMicrovmAuthToken`, for HTTP delivery. Grant it unless every input is small and no config sets a `path`. -- `lambda:PassNetworkConnector` on the connectors that the MicroVM uses. `RunMicrovm` authorizes each connector separately. With the defaults, that is `arn:aws:lambda::aws:network-connector:aws-network-connector:*`. -- `iam:PassRole` on the MicroVM execution role. -- `lambda:GetMicrovm`, for HTTP delivery, from `microvm` and from a session. A job over HTTP checks the MicroVM state after an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED` or a request without an answer. Without `lambda:GetMicrovm`, a job whose request gets that answer, or no answer, fails at once with `MicrovmDeliveryError`, and the message names `lambda:GetMicrovm`. A role without `lambda:GetMicrovm` fails a job whose request gets no answer, even when the MicroVM runs, for example a worker that takes more than 10 seconds to listen. See [The state check](#the-state-check). - -The function's role does not need `lambda:ResumeMicrovm`. The SDK does not resume a MicroVM. The service resumes it on the next request. - -The MicroVM execution role (`execution_role_arn`) needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the function ARN (`arn:aws:lambda:::function::*`). - -## `microvm_session`: run several jobs in one MicroVM - -`microvm_session` launches one MicroVM and runs a handler that sends jobs to it. The jobs share the MicroVM's files and processes. So a later job can use what an earlier job wrote. The MicroVM is terminated when the handler returns or raises. +A session keeps one MicroVM for several jobs. A later job can use the files that an earlier job wrote. The session terminates the MicroVM when its handler returns or raises. ```python import os @@ -195,20 +69,21 @@ SESSION = MicrovmSessionConfig( def pipeline(vm: MicrovmSession, ctx: DurableContext) -> Any: build = vm.invoke( ctx, - "clone-build", + "build", job_input={"repo": "org/app"}, options=MicrovmInvokeOptions( - path="/clone-build", timeout=Duration.from_minutes(15), heartbeat_timeout=Duration.from_minutes(2), ), ) - ctx.wait(Duration.from_minutes(5), name="cool-down") return vm.invoke( ctx, "test", job_input=build, - options=MicrovmInvokeOptions(timeout=Duration.from_minutes(10)), + options=MicrovmInvokeOptions( + timeout=Duration.from_minutes(10), + heartbeat_timeout=Duration.from_minutes(2), + ), ) @@ -217,50 +92,58 @@ def handler(event: dict, context: DurableContext) -> Any: return microvm_session(context, "pipeline", config=SESSION, handler=pipeline) ``` -The operation runs its durable operations in one child context named `name`: +Pass `vm.invoke` the context that the session handler receives, or a context nested in it. Set a `heartbeat_timeout` on every job. A job that still runs when the MicroVM ends then fails at its heartbeat timeout. -``` -pipeline child context -├── pipeline.session step: generates the session ID -├── pipeline.launch step: RunMicrovm -├── clone-build child context, one per job -│ ├── clone-build.callback callback -│ └── clone-build.request step: POSTs the job -├── cool-down wait -├── test -│ ├── test.callback -│ └── test.request -└── pipeline.terminate step: TerminateMicrovm +### Send a job to a named route + +This code runs inside the quick start's `handler`: + +```python +import dataclasses + +from aws_durable_execution_sdk_python_extras.microvm import MicrovmRequestConfig + +config = dataclasses.replace(CONFIG, request=MicrovmRequestConfig(path="/clone-build")) +result = microvm(context, "build", job_input={"repo": "org/app"}, config=config) ``` -- The client token of the launch is the SHA-256 hex digest of the session ID. The session ID is random, and it is checkpointed. So a retried or replayed launch returns the same MicroVM. -- Every job goes over HTTP. The MicroVM gets the `ALL_INGRESS` connector. A job without a `path` goes to the worker's `handler`. -- The first job is sent right after the launch. The MicroVM endpoint refuses requests until the `run` hook returns, and the request step retries a refused request inside its attempt. So the session needs no separate readiness wait. -- `vm.invoke` takes the context that the call runs in, as `microvm` does. Inside a `map` or `parallel` item, pass the item's context. A job must be created inside the session's child context. A job created in the context that `microvm_session` was called with runs, and the next replay then fails the execution with `NonDeterministicExecutionError`. So `vm.invoke` raises `TypeError` for that context. The core SDK does not check other contexts outside the session. -- `config.timeout` bounds the MicroVM's lifetime: it ends at `timeout` plus 5 minutes, at most 8 hours. Each job waits for its own `timeout`. So set a `heartbeat_timeout` on every job. A job still running when the MicroVM ends then fails at its heartbeat timeout. +In a session, set `MicrovmInvokeOptions(path="/clone-build", ...)` instead. Register the same path in the worker's `routes`. -A failed `vm.invoke` raises its MicroVM error inside the handler. The handler can catch it and send another job to the same MicroVM. An error that leaves the handler ends the session: +### Handle a failed job -1. A MicroVM error that the handler raises itself reaches the caller with its own class, such as `MicrovmJobFailedError`. -2. Any other error reaches the caller as the SDK's `ChildContextError`, as from `run_in_child_context`. This includes a MicroVM error that leaves a child context that the handler created: that child context records `ChildContextError` as its error type. The caller then gets the MicroVM error's message, but not its class. Inside the handler, the `ChildContextError` of that child context has the MicroVM error as its `__cause__`. An SDK error that leaves the handler, such as the `StepError` of a failed step, also reaches the caller as `ChildContextError`, with the SDK error as its `__cause__`. This rule matches `run_in_child_context`. The JavaScript operation differs here: it passes an SDK error to the caller with its own class. +Every MicroVM failure is a `MicrovmError`. Catch a subclass for one kind of failure. This code runs inside the quick start's `handler`: + +```python +from aws_durable_execution_sdk_python_extras.microvm import ( + MicrovmError, + MicrovmJobFailedError, + MicrovmTimeoutError, +) -An invalid job raises `ValidationError`, `TypeError`, or `ValueError` from `vm.invoke`, before the job's durable operations start. The handler can catch it too. A call without the context, `vm.invoke("build", job_input=..., options=...)`, raises Python's `TypeError`. A first argument that is not a durable context, as in `vm.invoke("build", "build", ...)`, raises `AttributeError`. +try: + result = microvm(context, "build", job_input={"repo": "org/app"}, config=CONFIG) +except MicrovmJobFailedError: + ... # The job handler in the MicroVM raised an error. +except MicrovmTimeoutError: + ... # No result or no heartbeat in time. A crashed MicroVM ends this way. +except MicrovmError: + ... # The launch or the delivery failed. +``` -The handler's return value is checkpointed as the session's result. So it must be serializable by the SDK, and it counts toward the checkpoint size limit. +A `microvm` call inside your own child context, `map` item, or `parallel` item reaches the code around it as `ChildContextError`. Its `__cause__` is the MicroVM error. -### Suspending an idle MicroVM +## Advanced -A running MicroVM pays compute charges. A suspended MicroVM pays only for snapshot storage, and it keeps its memory and files. A session can suspend its MicroVM between jobs in two ways: +### Suspend an idle MicroVM -1. `auto_suspend_on_idle=True` asks the worker in the MicroVM to suspend its own MicroVM when no job has run for `auto_suspend_idle_time`. The default idle time is 60 seconds, or the session `timeout` if that is shorter. The idle time must be at least 10 seconds. So a session whose `timeout` is under 10 seconds does not suspend by default, even with `auto_suspend_on_idle=True`. -2. `suspend_after=True` on a job makes the durable function suspend the MicroVM after that job. See [Suspending after a job](#suspending-after-a-job). +A suspended MicroVM pays only for snapshot storage, and keeps its memory and files. Suspending is off by default. -To suspend a MicroVM, set `idle_policy` with `"autoResumeEnabled": True`. The SDK does not resume a MicroVM; the service resumes it on the next request: +The SDK never calls `ResumeMicrovm`. The service resumes a suspended MicroVM on the next request, but only if the MicroVM's idle policy sets `autoResumeEnabled`. Setting the idle policy is your job. So every session that suspends must set `idle_policy` with `"autoResumeEnabled": True`. Without it, every job after a suspend fails with `MicrovmDeliveryError`. ```python -config = MicrovmSessionConfig( - image_identifier=IMAGE, - execution_role_arn=ROLE, +SESSION = MicrovmSessionConfig( + image_identifier=os.environ["MICROVM_IMAGE_ARN"], + execution_role_arn=os.environ["MICROVM_ROLE_ARN"], timeout=Duration.from_hours(1), auto_suspend_on_idle=True, idle_policy={ @@ -272,39 +155,14 @@ config = MicrovmSessionConfig( ) ``` -- Set `maxIdleDurationSeconds` above `maximumDurationInSeconds` (the session `timeout` plus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job. -- Without `autoResumeEnabled`, a job after a suspend fails after its retries with `MicrovmDeliveryError`. With the default retry strategy, that takes about 6 minutes. -- Set `suspendedDurationSeconds` to the session `timeout` or more. What happens to a job when that time ends is not verified. -- A job that arrives more than about 5 seconds after a suspend takes about 1 s longer. A job sent right after a suspend can take up to about 10 seconds, as in the next list. -- The idle policy needs all three fields, and `maxIdleDurationSeconds` must be at least 60. `RunMicrovm` rejects any other policy, and the launch fails with `MicrovmLaunchError`. The session does not check the policy itself. - -A job reaches a suspended MicroVM this way: - -1. For about 5 seconds after a suspend, the endpoint answers a request with HTTP 502 and `x-aws-proxy-error: MICROVM_CONNECT_FAILED`. That request does not resume the MicroVM. The request step retries it, after 250 milliseconds at first, doubling up to 4 seconds. Its 5th retry starts 7.75 seconds after the first request, so the retries reach past that window. With a `retry_window` under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds. -2. After that window, a request resumes the MicroVM. The endpoint holds the request for about 750 milliseconds, and then the worker answers it. -3. After each 502 with `MICROVM_CONNECT_FAILED`, the request step checks the state, at most once per 2 seconds. A `SUSPENDING` or `SUSPENDED` MicroVM keeps the job retrying. See [The state check](#the-state-check). -4. A request that arrives while the MicroVM suspends can get no answer. That request reaches the worker after the next resume. The worker ignores a second request for a callback ID that runs or recently ended. So the job runs once. - -The worker reads its idle time once, from the run hook payload of the launch. `RunMicrovm` sets the idle policy once, at the launch. So a deploy that changes `auto_suspend_on_idle` or `idle_policy` does not change what a running MicroVM does. - -An earlier release resumed a suspended MicroVM itself, and recorded `resumeBeforeDelivery` in the launch checkpoint. An execution in flight that launched with that release replays its launch checkpoint with `resumeBeforeDelivery: true`. This release ignores the field. So the next job of that execution no longer resumes the MicroVM. That job works only if the MicroVM was launched with `autoResumeEnabled: True`. A session that suspended its MicroVM with `suspend_after` and has no `idle_policy` is affected too: its next job after the suspend no longer resumes the MicroVM and fails after its retries with `MicrovmDeliveryError`. - -A MicroVM lives at most its `timeout` plus 5 minutes, and never longer than 8 hours. Every job over HTTP, of a session or of `microvm`, reports a MicroVM that ended: - -- A terminated MicroVM's endpoint answers HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, measured in us-east-1. After that answer, [the state check](#the-state-check) reports the state, and the job fails with `MicrovmNotRunningError`, a subclass of `MicrovmDeliveryError`, after one request. A retry finds the same state, so the job is not retried. The message ends with the request ID of the call that reported the state, for example `GetMicrovm request ID: .`. -- Every session, and every `microvm` job over HTTP, calls `CreateMicrovmAuthToken` before the first request, and again after a 401 or 403. The API model lists `ResourceNotFoundException` for that call. That answer fails the job at once with `MicrovmNotRunningError`, whose message ends with `CreateMicrovmAuthToken request ID: .`. For a terminated MicroVM whose record still exists, the call issues a token, measured in us-east-1. The live service's answer for a removed MicroVM is not verified. - -Suspending is off by default, because it needs more settings and changes what the MicroVM does between jobs: +- `auto_suspend_on_idle=True` makes the worker suspend its MicroVM after 60 seconds without a job. `auto_suspend_idle_time` changes the time. +- Set `maxIdleDurationSeconds` above the MicroVM's `maximumDurationInSeconds`, which is the session `timeout` plus 5 minutes. A shorter value can suspend a running job. +- The idle policy needs all three fields. `maxIdleDurationSeconds` must be at least 60. +- Enable the `suspend` and `resume` hooks in the image. -- `idle_policy` must set `autoResumeEnabled: True`, as above. -- With `auto_suspend_on_idle`, the MicroVM's role needs `lambda:SuspendMicrovm` on the image. That permission lets code in any MicroVM of the image suspend any other. So use one image per trust boundary. -- Background processes that an earlier job started stop while the MicroVM is suspended. The worker counts only its running jobs. +### Suspend after a job -The worker must support `auto_suspend_on_idle`. The JavaScript worker [`@aws/durable-execution-sdk-js-microvm-worker`](https://github.com/aws/aws-durable-execution-sdk-js/tree/main/packages/aws-durable-execution-sdk-js-microvm-worker) and the Python worker [`aws-durable-execution-sdk-python-microvm-worker`](../aws-durable-execution-sdk-python-microvm-worker) both do. Enable the `suspend` and `resume` hooks in the image. The worker then refuses jobs while the MicroVM freezes, and accepts them again as soon as it resumes. Without the `resume` hook, the first job after a resume can wait a little over 30 seconds, until the worker's refusal ends and the request step delivers again. - -### Suspending after a job - -`MicrovmInvokeOptions(suspend_after=True)` makes the durable function suspend the MicroVM after this job. It is an option of one job, not of the session. `microvm` raises `ValidationError` for it, because `microvm` terminates its MicroVM after its one job. A value that is not a `bool` raises `ValidationError` too, before the job's durable operations start. +`MicrovmInvokeOptions(suspend_after=True)` makes the durable function suspend the MicroVM after that job. Use it on the job before a long wait, such as a callback. This code runs inside a session handler, such as `pipeline` above: ```python build = vm.invoke( @@ -317,7 +175,7 @@ build = vm.invoke( suspend_after=True, ), ) -approval = ctx.create_callback(name="approval").result() # a long wait +approval = ctx.create_callback(name="approval").result() # A long wait. vm.invoke( ctx, "deploy", @@ -329,141 +187,48 @@ vm.invoke( ) ``` -What happens: - -1. The session calls `SuspendMicrovm` with the session's MicroVM ID when the job's callback result arrives, whether the job succeeded or failed. The call runs inside the job's child context, before the child context returns. A wait for the callback does not suspend the MicroVM. -2. It adds no step and no checkpoint. The child context's own completion checkpoint covers it. On replay, a completed child context does not run again, so the session sends no second suspend. -3. A failed `SuspendMicrovm` call logs a warning and does not fail the job. The MicroVM then keeps running until the idle suspend or the end of the session. -4. The job request carries no field for it, and the worker does nothing for it. The worker needs no change, and the MicroVM's role needs no suspend permission. -5. The next job's first request gets HTTP 502, and the service resumes the MicroVM on a later request. So the session needs the `idle_policy` of [Suspending an idle MicroVM](#suspending-an-idle-microvm), with `autoResumeEnabled: True`. - -The setting works with or without `auto_suspend_on_idle`. With both, the function suspends right after the job, and the worker's idle time applies after each resume. - -The call is not a durable operation, so these rules are yours to follow. The session adds no guard for them: - -1. Do not use it on a job that runs next to other jobs of the session (`map`, `parallel`, promise combinators). Those jobs freeze until a later request makes the service resume the MicroVM, and can hit their heartbeat timeout. -2. The suspend runs inside the job's child context, so it follows the child context replay rules. A result larger than the child context checkpoint limit makes the child context run again on replay, and each run suspends again. -3. A retry after the Lambda invocation fails can send the suspend again. The invocation can also end while the call runs, when other operations wait. The call is then sent again. Suspending a suspended MicroVM succeeds. -4. Adding or removing `suspend_after` changes what the session does on replay. Change it only through a new function version. - -Use `suspend_after` only on the job before a long wait, such as a durable wait or a callback: - -- Not on back-to-back jobs. Each suspend writes a snapshot, and each resume adds about 1 second before the next job, or up to about 10 seconds when the job is sent right after the suspend. -- Not on `map` items or other jobs that run at the same time. The suspend freezes the jobs that still run. -- Not on the last job. The session terminates the MicroVM after its handler returns, so the snapshot is wasted. - -It needs: - -1. The `resume` hook in the image. Without it, the worker refuses the next job with HTTP 503 for up to 30 seconds after the suspend. -2. A `heartbeat_timeout` on the next job of at least its `retry_window` plus 30 seconds. The heartbeat timeout starts before delivery, so it includes the resume and the refusal. -3. An `idle_policy` with `autoResumeEnabled: True`, as in [Suspending an idle MicroVM](#suspending-an-idle-microvm). -4. `lambda:SuspendMicrovm` on the image in the function's role. The MicroVM's role needs `lambda:SuspendMicrovm` only for `auto_suspend_on_idle`. - -The suspend runs after a job timeout or a heartbeat timeout too, so a job that still runs in the MicroVM is frozen, not stopped. - -The call ends at the invocation reserve, like the other calls of the session. A failed suspend, or a call that does not end in time, is not retried. The MicroVM may then still be running until the idle suspend or the end of the session. The warning names `lambda:SuspendMicrovm` for a missing permission. - -### The state check - -Every job over HTTP, of a session or of `microvm`, checks the MicroVM state after a sign that the MicroVM does not run. The check finds a MicroVM that ended. It does not resume a MicroVM. - -1. The sign is an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, or a request without an answer. A booting MicroVM's endpoint holds the request until the worker listens. A boot gives no such sign, unless the worker takes more than 10 seconds to start listening. Then the first delivery gets no answer, and the operation makes one GetMicrovm call. -2. The request step calls `GetMicrovm` after each such answer, at most once per 2 seconds. The call runs first, before the `retry_window` and the backoff are checked, and only the invocation reserve limits it. So a delayed answer that uses up the window still gets the check. With less than 1 second left before the reserve, no call starts, and the attempt ends with `MicrovmEndpointUnavailableError`, which the default retry strategy retries. -3. `TERMINATING`, `TERMINATED`, or not found fails the job at once with `MicrovmNotRunningError`. -4. `RUNNING`, `PENDING`, `SUSPENDING`, or `SUSPENDED` keeps the job retrying. The service resumes a suspended MicroVM on a request, when its idle policy sets `autoResumeEnabled`. -5. A denied `GetMicrovm` fails the job at once with `MicrovmDeliveryError`. A retry is denied the same way, so the default retry strategy does not retry it. The message names `lambda:GetMicrovm` and ends with the `GetMicrovm` request ID. -6. Any other failed `GetMicrovm`, such as throttling, a 5xx answer, a timeout, or a connection error, logs one warning per step attempt, so up to 5 with the default retry strategy: `GetMicrovm failed, so the operation could not check whether the MicroVM has ended. The request step keeps sending the job.` The request step then keeps sending the job. +- The session needs the same `idle_policy` with `"autoResumeEnabled": True`. +- The suspend is not a durable operation, and the session adds no guard around it. So parallel operations on the same session are your responsibility. A job that runs next to the suspended job (`map`, `parallel`) freezes with the MicroVM and can reach its heartbeat timeout. +- Give the next job a `heartbeat_timeout` of at least its `retry_window` plus 30 seconds. +- The function's role needs `lambda:SuspendMicrovm` on the image. +- Change `suspend_after` only through a new function version. -### Session permissions +### Retries -A session needs the permissions of `microvm` with HTTP delivery: `lambda:RunMicrovm`, `lambda:TerminateMicrovm`, `lambda:CreateMicrovmAuthToken`, `lambda:PassNetworkConnector`, and `iam:PassRole`. Suspending adds the permissions listed above: `lambda:SuspendMicrovm` in the function's role for `suspend_after`, and in the MicroVM's role for `auto_suspend_on_idle`. Grant `lambda:GetMicrovm` to every session's function role, as for `microvm` with HTTP delivery. The role does not need `lambda:ResumeMicrovm`. `lambda:GetMicrovm` is required. Without it, a job whose request gets an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, or no answer, fails at once, with a message that names `lambda:GetMicrovm`. The check runs after any such answer, at most once per 2 seconds. +The operations retry throttling, server errors, and network errors with `default_microvm_retry_strategy`. Set `retry_strategy` in the config to replace it. In a custom strategy, match an AWS error by `error.response["Error"]["Code"]`, not by its message. -### Subtypes +### Limits and timeouts -The JavaScript operations record MicroVM subtypes, such as `Microvm` on the child context, `MicrovmSession` on a session, and `MicrovmLaunch` on the launch step. This package records the SDK's default subtypes for now, because the Python SDK does not accept custom subtypes yet. +- `timeout` is between 1 second and 8 hours. The MicroVM lives at most `timeout` plus 5 minutes. +- `job_input` must be JSON-serializable and at most `MAX_JOB_INPUT_BYTES` (6 MiB minus 8 KiB) as UTF-8 JSON. Pass larger data by reference, for example as an S3 key. +- A small `microvm` job goes in the `RunMicrovm` request. A large job, a job with `request.path`, and every session job go over HTTP. A job over HTTP needs a function timeout of at least `retry_window` plus 11 seconds: 71 seconds by default. +- `heartbeat_timeout` starts before the launch. So it must cover the launch, the boot, and the delivery. -On replay, the SDK compares the subtype of each operation with its checkpoint. A mismatch fails the execution with `NonDeterministicExecutionError`. So a release that changed the subtypes would fail every execution that waits on a MicroVM during the deploy. So this package is published only after the SDK accepts custom subtypes, and it records the MicroVM subtypes from its first release. +### Deploys -## Best practices for MicroVM operations +Invoke the durable function through a published version or an alias, not `$LATEST`. An execution on `$LATEST` replays on new code after a deploy. A launch that retries on new code can fail and leave a MicroVM running until its maximum duration. -The MicroVM operations handle the network, the service, and the worker lifecycle. They do not check every setting. This section lists what your code and your deploy must do. Each rule states what happens if you break it. - -### Deploys and versions - -1. Invoke the durable function through a published version or an alias, not `$LATEST`. - - An execution started through a version or an alias is pinned to that version. Every replay runs the same code and the same configuration, including the environment variables. - - An execution started through `$LATEST` is not pinned. After a deploy, it resumes on the new code. See [Invoking durable Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking.html). - - So under `$LATEST`, a launch step that retries after a deploy can fail with `MicrovmLaunchError`. A MicroVM that an earlier attempt of that step created can then keep running, and be billed, until its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. -2. Do not read MicroVM settings from a source that can change during an execution, such as an SSM parameter, AppConfig, or an S3 object. Code outside a step runs again on every replay. So it reads the value again. A value that changes between two attempts of a launch step can fail the launch, as in rule 1. Read such a value inside a step. A replay then uses the checkpointed value. -3. The MicroVM packages are experimental. A release can change the durable operations that the MicroVM operations record. [Subtypes](#subtypes) describes one such change. An execution that replays its history on such a release fails with `NonDeterministicExecutionError`. So let executions that started on an earlier build end on that build. Under rule 1, this needs no extra step: a new version starts only new executions, and running executions continue on their own version. Under `$LATEST`, let the running executions end before you deploy a new build. - -### Function timeout - -1. Set the function timeout so that each request step starts with at least 11 seconds left in the invocation. With less time left, the step attempt fails at once. A function timeout under 11 seconds fails every attempt. So every job over HTTP fails with `MicrovmDeliveryError`, and its message says to raise the function timeout. Every session job goes over HTTP. [Defaults and limits](#defaults-and-limits) explains the 11 seconds and gives the recommended timeout. - -### Job inputs - -1. Pass a JSON-serializable input with finite numbers. Any other input raises `TypeError` before any durable operation starts. See [Errors](#errors). -2. Keep the job input's JSON at or under `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, in UTF-8. Pass larger data by reference, for example as an S3 key. - - The Python worker reads a job request body of at most 6 MiB. The body holds the input's JSON and a few job fields. - - `microvm` and `vm.invoke` measure the input's JSON before any durable operation starts. A larger input raises `ValueError`. So `microvm` launches no MicroVM for it. -3. Expect a job to go over HTTP in three cases: - - The `run` hook payload of a `microvm` job is over 4,096 code points. The payload holds the input, the callback ID, and a few other fields. - - The `microvm` config sets `request.path`. The job then goes over HTTP for any input size. - - The job is a session job. Every session job goes over HTTP. - - A job over HTTP needs four things that a `run` hook job does not: - - `lambda:CreateMicrovmAuthToken` in the function's role. Without it, the job fails with `MicrovmDeliveryError`. See [Permissions](#permissions). - - The function timeout of [Function timeout](#function-timeout). With a shorter timeout, the job fails with `MicrovmDeliveryError`. - - Ingress to the MicroVM. The operation gives such a MicroVM the `ALL_INGRESS` connector. If you set `ingress_network_connectors`, include a connector that allows ingress. Without ingress, the request step cannot reach the MicroVM's endpoint. So the job fails. - - `lambda:PassNetworkConnector` on the `ALL_INGRESS` connector in the function's role. Its ARN is `arn:aws:lambda::aws:network-connector:aws-network-connector:ALL_INGRESS`. `RunMicrovm` authorizes each connector separately. So a role that allows `NO_INGRESS` and `INTERNET_EGRESS`, but not `ALL_INGRESS`, fails every HTTP launch with `MicrovmLaunchError`. - - A `microvm` job whose payload fits the `run` hook, and whose config has no `request.path`, needs none of these four. So a function that passed its tests with such jobs can fail every job over HTTP. See [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). - -### Job handlers - -1. Follow [Best practices for job handlers](../aws-durable-execution-sdk-python-microvm-worker/README.md#best-practices-for-job-handlers) in the worker README for the code that runs each job in the MicroVM. -2. Make the side effects of a job safe to repeat. - - The worker ignores a second delivery of a callback ID. It remembers the callback IDs of its last 1,000 finished jobs, and it forgets them when its process restarts. - - Each `microvm` call and each `vm.invoke` creates a new callback ID. So when your code runs a job again after a failure, the worker runs it again, also in the same MicroVM. - - A failure can follow a run that already changed something. After `MicrovmLaunchError` with `run` hook delivery, the MicroVM can still run the job. An HTTP launch request contains no job. The request step starts only after the launch succeeds. So when you invoke through a version or an alias, no MicroVM runs the job after `MicrovmLaunchError` with HTTP delivery. After `MicrovmTimeoutError`, the job can have done part of its work. See [Errors](#errors). - -### The HTTP job route - -1. Answer the job request itself with a 2xx status at once, and run the job in the background. The worker package does this for its routes. - - The request step waits at most 10 seconds for an answer, and then sends the job again. So a route that answers only when a longer job ends receives the job again and again. The delivery then fails with `MicrovmDeliveryError`, after the retries of both tiers. - - The operation does not follow redirects. It fails the job at once with `MicrovmDeliveryError` for a 3xx answer, and for a 4xx answer other than 401, 403, and 429. A proxy in front of the route that redirects, for example to add a trailing slash, causes the same failure. See [Retries in two tiers](#retries-in-two-tiers). -2. Ignore a second request for a callback ID that the route already ran. The same job can arrive more than once. A route served without the worker package must do this itself, or it runs the job again. See [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). -3. Give `path` a value that follows the path rules in [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). The same check rejects a query, a fragment, and a path that starts with `/aws/lambda-microvms/`, where Lambda sends the lifecycle hooks. An invalid path raises `ValidationError` when `microvm` or `vm.invoke` is called, before its durable operations start. Register the same string as the worker's route. The worker answers a path that it does not serve with HTTP 404, and the operation then fails the job at once with `MicrovmDeliveryError`. - -### Sessions +### Permissions -1. Call `vm.invoke(ctx, name, job_input=..., options=...)` with the context that the session handler receives, or a context nested in it. A job created in another context runs, and the next replay fails the execution with `NonDeterministicExecutionError`. `vm.invoke` raises `TypeError` only for the context that `microvm_session` was called with. It does not detect any other wrong context. See [`microvm_session`](#microvm_session-run-several-jobs-in-one-microvm). -2. Pass a function as `handler`. `microvm_session` does not check it before it starts. A value that is not callable raises Python's `TypeError` when the session calls it, after the launch. The session then terminates the MicroVM, and the caller receives `ChildContextError`. So the mistake costs one MicroVM boot. mypy reports it before that. -3. If you set `auto_suspend_idle_time`, meet three conditions: - - `auto_suspend_on_idle` is `True`. - - The idle time is at least 10 seconds. - - The idle time is at most the session `timeout`. +The function's execution role needs: - A value that breaks any one condition raises `ValidationError` when `microvm_session` is called, before any durable operation starts. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). -4. Set `suspend_after` only on the job before a long wait. Never set it on a job that runs next to other jobs of the session. Add `lambda:SuspendMicrovm` to the function's role, set an `idle_policy` with `autoResumeEnabled: True`, and change `suspend_after` only through a new function version. Give the next job a `heartbeat_timeout` of at least its `retry_window` plus 30 seconds. On back-to-back jobs, each suspend and resume adds latency and a snapshot. A shorter heartbeat timeout can fail the next job with `MicrovmTimeoutError` while the MicroVM resumes. See [Suspending after a job](#suspending-after-a-job). -5. A session that suspends its MicroVM, with `auto_suspend_on_idle` or `suspend_after`, needs an `idle_policy` with `autoResumeEnabled: True` and a `maxIdleDurationSeconds` above the session `timeout` plus 5 minutes. Without `autoResumeEnabled`, every job after a suspend fails with `MicrovmDeliveryError`. A shorter `maxIdleDurationSeconds` suspends a running job. Change `auto_suspend_on_idle` or `idle_policy` only through a new version, as in [Deploys and versions](#deploys-and-versions). The launch sets both, so a deploy changes them only for sessions that launch after it. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). +- `lambda:RunMicrovm` and `lambda:TerminateMicrovm` on the MicroVM image. +- `lambda:CreateMicrovmAuthToken` and `lambda:GetMicrovm`, for jobs over HTTP. +- `lambda:PassNetworkConnector` on the network connectors, and `iam:PassRole` on the MicroVM role. +- `lambda:SuspendMicrovm` on the image, for `suspend_after`. -### Retry strategies +The function's role does not need `lambda:ResumeMicrovm`. -1. Keep `default_microvm_retry_strategy` unless you need another policy. It decides by the error class, the error code, and the HTTP status. A strategy that retries less fails a job on a throttle or a network error. A strategy that retries more repeats errors that a retry cannot change, such as `AccessDeniedException`, and the job fails later. See [Retries in two tiers](#retries-in-two-tiers). -2. In a custom `retry_strategy`, match an AWS error by its code, `error.response["Error"]["Code"]`. Match an error of the operation by its class, such as `MicrovmEndpointUnavailableError`. The API model names each error code. It defines each message only as a string. So the service can change a message without an API change. A strategy that matches the message then decides differently, and nothing reports the change. +The MicroVM's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the function ARN. With `auto_suspend_on_idle`, it also needs `lambda:SuspendMicrovm` on the image. -### Custom clients and tests +## Reference -1. A `client` that you pass must behave as a boto3 `lambda-microvms` client. - - It returns outputs in the boto3 shape, with `ResponseMetadata`. A `RunMicrovm` output without `microvmId`, or without `endpoint` for HTTP delivery, fails the launch with `MicrovmLaunchError`. An output without `ResponseMetadata.RequestId` gives error messages without a request ID. - - It raises botocore `ClientError` with a `response` that holds `Error.Code` and `ResponseMetadata.HTTPStatusCode`. The default strategy reads both. It fails the step at once for an error of any other class, unless the error is a timeout or a connection failure. - - It sets connect and read timeouts, such as the 5 and 30 seconds of the default client. botocore's own defaults are 60 seconds each. A `RunMicrovm` or `TerminateMicrovm` call that runs past the function timeout records no outcome for its step attempt. So the step runs again. A `CreateMicrovmAuthToken` or `GetMicrovm` call that reaches the invocation reserve fails its attempt. Its thread keeps running until the call returns. So the call can still reach the service after the step stopped waiting for it. -2. Pass an `opener` when the network reaches the MicroVM endpoint only through a proxy. The default opener ignores the proxy settings in the environment. So without an `opener`, the launch succeeds, and every POST fails. The job then fails with `MicrovmDeliveryError`, after the retries of both tiers. See [Retries in two tiers](#retries-in-two-tiers). -3. In tests, wrap a real `lambda-microvms` client in botocore's `Stubber`. `Stubber` checks each stubbed output against the API model. Its `add_client_error` raises `ClientError` with `Error.Code` and `ResponseMetadata.HTTPStatusCode`. So the operation and the default strategy read the same fields as from the service. A fake with another shape tests paths that the service never produces. Add `ResponseMetadata` with a `RequestId` to a stubbed output, or pass `response_meta` to `add_client_error`, to test the request IDs in messages. moto has no `lambda-microvms` backend, as of moto 5.2.4. +- [docs/microvm.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-extras/docs/microvm.md): job delivery, the durable operations each call records, all errors, the retry tiers, the state check, every permission, and best practices. +- [MicroVM worker](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-microvm-worker): the code that runs in the MicroVM. +- [Source](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm): config fields and docstrings. +- [`@aws/durable-execution-sdk-js-extras`](https://github.com/aws/aws-durable-execution-sdk-js/tree/main/packages/aws-durable-execution-sdk-js-extras): the JavaScript counterpart. +- [Lambda MicroVMs guide](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html). -### Permissions and monitoring +## License -1. Give the function's execution role every permission in [Permissions](#permissions), including `lambda:TerminateMicrovm`. Without it, every operation still returns its result, and every MicroVM runs, and is billed, until its maximum duration. -2. Create an alarm on the log line that starts with `MicroVM "": terminate failed for `, for example with a CloudWatch Logs metric filter that matches `terminate failed for`. A failed terminate does not fail the operation. The operation logs this error instead. The MicroVM then keeps running, and is billed, until its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. This line is the only sign. See [What it does](#what-it-does). +This project is licensed under the [Apache-2.0 License](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/LICENSE). diff --git a/packages/aws-durable-execution-sdk-python-extras/docs/microvm.md b/packages/aws-durable-execution-sdk-python-extras/docs/microvm.md new file mode 100644 index 000000000..22dc022f3 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/docs/microvm.md @@ -0,0 +1,466 @@ +# MicroVM operations: reference + +This file describes in detail what `microvm` and `microvm_session` do. +The [README](../README.md) shows how to use them. + +## `microvm`: run a job in an AWS Lambda MicroVM + +`microvm` launches a new [Lambda MicroVM](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html), delivers the job to it, waits for the job's result, and terminates the MicroVM. The function invocation ends while the MicroVM works, so no Lambda compute is billed during the wait. + +```python +import os + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import DurableContext +from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python_extras.microvm import MicrovmConfig, microvm + +CONFIG = MicrovmConfig( + image_identifier=os.environ["MICROVM_IMAGE_ARN"], + execution_role_arn=os.environ["MICROVM_ROLE_ARN"], + timeout=Duration.from_hours(1), + heartbeat_timeout=Duration.from_minutes(5), +) + + +@durable_execution +def handler(event: dict, context: DurableContext) -> dict: + return microvm(context, "build", job_input={"repo": event["repo"]}, config=CONFIG) +``` + +The first argument is the context that the call runs in. Inside a child context or a `map` item, pass that context, so the operation's steps belong to it. `job_input` and `config` are keyword-only. A job can be a string, and so is the name. So positional arguments in the wrong order would pass validation and run the wrong job. + +The code in the MicroVM receives the job and reports its result. The worker package [`aws-durable-execution-sdk-python-microvm-worker`](../../aws-durable-execution-sdk-python-microvm-worker) does this, and so does the JavaScript worker. The job document is the same in every SDK language. + +### How the job reaches the MicroVM + +The operation picks the delivery from the input size and the config: + +1. A `microvm` job goes in the `RunMicrovm` request when its `run` hook payload fits in 4,096 Unicode code points and its config sets no `request.path`. The payload holds the input, the callback ID, and a few other fields. Lambda passes the payload to the worker's `run` lifecycle hook. The MicroVM gets the `NO_INGRESS` connector, so nothing can reach it. +2. Every other job goes over HTTP: + - A `microvm` job whose `run` hook payload is over 4,096 code points. + - A `microvm` job whose config sets `request.path`, whatever its size. + - Every session job. + + The launch request carries no job, and a request step POSTs the job to the MicroVM's endpoint after it starts. The Python worker accepts a request body of at most 6 MiB. The MicroVM gets the `ALL_INGRESS` connector, and every request carries a short-lived auth token that the operation creates. + +The worker package runs its `handler` for both deliveries. So the code in the MicroVM does not change with the input size. + +The HTTP delivery waits inside the Lambda invocation until the MicroVM's endpoint accepts the job, usually one to two seconds. The `run` hook delivery adds no Lambda time for the boot. + +An application that already serves HTTP routes can receive every job on its own route: + +```python +import dataclasses + +from aws_durable_execution_sdk_python_extras.microvm import MicrovmRequestConfig + +config = dataclasses.replace(CONFIG, request=MicrovmRequestConfig(path="/clone-build")) +microvm(context, "build", job_input=job_input, config=config) +``` + +A `path` sends every job over HTTP, whatever its size. The route must answer with a 2xx status at once and run the job in the background. `port` names a port other than 8080. `retry_window` sets how long the request step retries inside one attempt, 60 seconds by default. + +The same job can arrive again, also after it ended, so ignore a second request for a callback ID that you already ran. The worker package does this. A route served without it must do it itself. A job arrives again in these cases: + +1. A request reaches the route, and its answer is lost. +2. The request step stops waiting for a request after at most 10 seconds, but the request keeps running in its thread and reaches the route. The next attempt sends the job again. +3. The invocation ends after the route accepted the job, and before the request step checkpointed its success. The step then runs again in the next invocation. + +The path must be an RFC 3986 path without a `.` or `..` segment, and the operation sends it exactly as written. A dot segment is rejected also when the dot is written as `%2e`, because the JavaScript SDK's `fetch` resolves such a segment, and a proxy or server can too. So both SDKs accept the same paths. Percent-encode any other character, for example a space as `%20`. The workers match the raw request path, so register the same string as the worker's route. An invalid path raises `ValidationError` before any durable operation starts. + +### What it does + +The operation runs its durable operations in one child context named after the second argument: + +1. `.callback` creates a durable callback. The service generates the callback ID, and the SDK checkpoints it. So the ID is unique to this call and stays the same on replay. +2. `.launch` calls `RunMicrovm`. The client token is the SHA-256 hex digest of the callback ID. So a retried or replayed launch returns the same MicroVM instead of starting a second one. With `run` hook delivery, the launch request contains the job. The step checkpoints plain JSON, `{"microvmId": "...", "endpoint": "...", "delivery": "http"}`, as the JavaScript operation does. The `endpoint` is present only for HTTP delivery. So a reader of the execution history finds the MicroVM ID there. +3. With HTTP delivery, `.request` creates an auth token and POSTs the job. +4. The child context waits for the MicroVM to complete the callback. +5. `.terminate` calls `TerminateMicrovm`. It runs after success, failure, and timeout alike, and after a failed delivery. + +Replay follows the delivery that the launch checkpointed, not a new choice. A deploy between the launch and a replay can change the input or the config. The launch result comes from the checkpoint, so the job still goes the way the MicroVM was launched for. + +A terminate failure does not fail the operation. The job's outcome is already recorded at that point. So the operation logs an error instead of raising. The MicroVM then keeps running, and is billed, until the platform ends it at its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. + +The terminate step runs only when the operation sees the job's outcome. In these cases it never does, and the MicroVM also runs until its maximum duration: + +- A `map` or `parallel` completes early, for example with `min_successful`, while an item still waits for its MicroVM. The SDK never runs that item again. +- The execution fails, times out, or is stopped while the operation waits. +- A launch attempt that timed out or lost its connection can still have created a MicroVM. When the launch step still fails, the operation never learns that MicroVM's ID. So it cannot terminate that MicroVM. + +### Errors + +Every MicroVM failure is a `MicrovmError`. Catch the base class for all of them, or a subclass for one kind: + +| Error | When | +|---|---| +| `MicrovmLaunchError` | `RunMicrovm` failed after all retries, or the launch request was not valid, for example with an empty `image_identifier` or `execution_role_arn`. A MicroVM can still run the job. The list below the table says when. | +| `MicrovmDeliveryError` | HTTP delivery failed after all retries, or the route answered a 4xx status such as 404, or a redirect, or GetMicrovm was denied (the message names the permission). A job after a suspend fails this way after its retries when the session's `idle_policy` does not set `autoResumeEnabled`. | +| `MicrovmNotRunningError` | A `MicrovmDeliveryError`: the MicroVM is terminating, terminated, or removed. It is not retried. Any job over HTTP, from `microvm` or a session, reports it after an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED` or a request without an answer: the request step then calls `GetMicrovm` after each such answer, at most once per 2 seconds. The endpoint of a terminated MicroVM answers that 502, measured in us-east-1. The message names the state, the `stateReason`, and the `GetMicrovm` request ID. Any job over HTTP also reports it when `CreateMicrovmAuthToken` reports the MicroVM removed. A job in the `run` hook has no request step, so a MicroVM that ends before the job runs fails it with `MicrovmTimeoutError`. See [The state check](#the-state-check). | +| `MicrovmJobFailedError` | The job handler in the MicroVM failed, or the MicroVM sent a result that is not valid JSON. The message includes the error type that the MicroVM reported. | +| `MicrovmTimeoutError` | No result arrived within `timeout`, or no heartbeat within `heartbeat_timeout`. A MicroVM that crashed ends this way. | + +After `MicrovmLaunchError`, the delivery decides whether a MicroVM runs the job: + +1. With `run` hook delivery, the launch request contains the job. A launch attempt that timed out or lost its connection can still have created a MicroVM. That MicroVM then runs the job. So running the job again after this error can run it twice. +2. With HTTP delivery, the launch request contains no job. The request step starts only after a successful launch. So when you invoke through a version or an alias, no MicroVM runs the job after an HTTP launch fails. + +```python +from aws_durable_execution_sdk_python_extras.microvm import ( + MicrovmError, + MicrovmTimeoutError, +) + +try: + result = microvm(context, "build", job_input=job_input, config=CONFIG) +except MicrovmTimeoutError: + ... # The MicroVM crashed, or took too long. +except MicrovmError: + ... # Any other MicroVM failure. +``` + +The class is the same on the first run and on every replay, so code that branches on it stays deterministic. A `microvm` call inside your own child context, or inside a `map` or `parallel` item, reaches the code around that context as the SDK's `ChildContextError`. Its `__cause__` is the MicroVM error, so check `isinstance(error.__cause__, MicrovmTimeoutError)` there. The operation raises three errors before any durable operation starts. An invalid name or config raises the SDK's `ValidationError`. An input that is not JSON-serializable raises `TypeError`. An input over `MAX_JOB_INPUT_BYTES` raises `ValueError`. `Duration` raises the same `ValidationError` for a negative value. The operation does not check `image_identifier` and `execution_role_arn` before it starts. boto3 checks them against the API model before it sends `RunMicrovm`, and the service checks them again. So an invalid value fails the launch step at once with `MicrovmLaunchError`, after the callback step. + +A failed AWS call keeps its request ID in the error message, in the form ` request ID: .`, for example `RunMicrovm request ID: .`. A support case can quote it. This applies to `RunMicrovm`, `GetMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The SDK checkpoints the message of a failed step, and not its `__cause__`. So the request ID is in the message, the handler sees it on the first run and on replay, and the execution history keeps it. A failed `TerminateMicrovm` call does not fail the operation. So a `TerminateMicrovm` request ID appears only in the terminate log entry and in the terminate step record. A message has no request ID when the answer carries none, for example after a connection failure. A message also has no request ID when the error is not the answer of the Lambda MicroVMs call. For example, a `client` with role-assuming credentials calls STS `AssumeRole` inside each call, and an STS error carries the request ID of STS. The operation recognizes such an error by its `operation_name`. The step raises a copy of the botocore error, of the same class, with the same `response`, and with the request ID in its message. So `default_microvm_retry_strategy` decides as before. A custom `retry_strategy` receives the error that botocore raised, without the request ID. So a strategy that matches the message, such as `create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))`, decides on the message of the service. A `ResourceNotFoundException` from `GetMicrovm` or `CreateMicrovmAuthToken` is the exception. It means that the MicroVM no longer exists. So the operation raises `MicrovmStateError` instead, and the strategy receives that error. Its `__cause__` is the botocore error. An error that the operation creates itself, such as `MicrovmStateError` or "RunMicrovm returned no microvmId", has the request ID in its message when the strategy receives it. A custom client can raise one error object for several calls. Each step message then names only its own call, and the object keeps its message. + +When the error escapes the handler, the execution's error type is the class's fully-qualified name, such as `aws_durable_execution_sdk_python_extras.microvm.errors.MicrovmJobFailedError`. The SDK uses that name to rebuild the class on replay. The JavaScript operation reports the short name, `MicrovmJobFailedError`. So a rule that matches the error type of both SDKs, such as a Step Functions `Catch` or a metric filter, must match both forms. For example, match on the suffix. + +### Retries in two tiers + +A step retry costs one durable operation and one more Lambda invocation. Most failures here are short, so the steps retry in two tiers: + +1. **Inside one step attempt.** The boto3 client retries throttling and server errors of `RunMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The request step retries the POST itself: connection errors, timeouts, 429, and 5xx, with a delay from 250 milliseconds to 4 seconds. It retries one 401 or 403 with a new auth token. The endpoint answers this way until the MicroVM's `run` hook returns. This tier ends after `retry_window`, or 10 seconds before the invocation times out, whichever comes first. +2. **The step retry strategy.** When the first tier gives up, the step retries after a longer delay, outside the invocation. `default_microvm_retry_strategy` is the default, unless the config sets `retry_strategy`. It retries throttling, server errors, connection failures, timeouts, and the end of the first tier, for at most 5 attempts, with a delay of 2, 4, 8, and 16 seconds. Every other error, such as `ValidationException`, `AccessDeniedException`, or `InsufficientCapacityException`, fails the step at once. + +A route that answers another 4xx status fails the operation without a retry, with `MicrovmDeliveryError`. So does a redirect, 3xx. The operation does not follow redirects. The execution history stores the error message. A query in the `Location` can hold a secret. So the message names the path, and the status of a 3xx answer. It omits the `Location`, and the URL that a followed redirect reached. Set the path that the route serves, for example with its trailing slash. The JavaScript operation also fails every 3xx at once, with `MicrovmDeliveryError`. + +The POST ignores the proxy settings in the environment, as the JavaScript SDK's `fetch` does. boto3 reads `HTTPS_PROXY`. So in a network that reaches the MicroVM endpoint only through a proxy, the launch succeeds and every POST fails. Pass an `opener` in the config for that network, for example `urllib.request.build_opener(urllib.request.ProxyHandler({"https": "http://proxy.internal:3128"}))`. When an opener follows a redirect, the delivery still fails at once with `MicrovmDeliveryError`. The followed request carries no auth token, because the operation adds its headers as unredirected headers. + +The 10-second reserve keeps time for the step to record its failure. A Lambda timeout during a step attempt records no outcome for the attempt. So every call of the first tier must end before the reserve. boto3 cannot cancel a call, so a `CreateMicrovmAuthToken` call runs in a separate thread, and the step stops waiting for it at the reserve. Each POST runs in a separate thread too, and the step waits for it at most 10 seconds, or until the reserve. A socket timeout alone would not bound it, because each byte received starts the socket timeout again. + +### Defaults and limits + +- `timeout` must be between 1 second and 8 hours, because a MicroVM lives at most 8 hours. The MicroVM's `maximumDurationInSeconds` is `timeout` plus 5 minutes, at most 8 hours. So a timeout above 7 hours 55 minutes leaves the MicroVM less than 5 minutes beyond the callback's timeout. +- `job_input` is sent as JSON. So the MicroVM receives what `json.dumps` writes: a tuple arrives as a list, and a dict key that is not a string arrives as a string. +- The worker accepts a job request body of at most 6 MiB. The body holds the JSON of `job_input` and a few job fields. The job fields take less than 3 KiB. So `job_input` can be at most `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, as UTF-8 JSON. A larger input raises `ValueError` from `microvm` or `vm.invoke` before any durable operation starts. +- The MicroVM gets the `INTERNET_EGRESS` connector, because it needs egress to reach the Lambda callback APIs. For ingress, it gets `NO_INGRESS` with `run` hook delivery and `ALL_INGRESS` with HTTP delivery. If you set `ingress_network_connectors`, allow ingress for every job that goes over HTTP. A `microvm` job goes over HTTP when its `run` hook payload is over 4,096 code points, or when its config sets `request.path`. Every session job goes over HTTP. An empty tuple sends no connector. +- `heartbeat_timeout` starts when the callback is created, before the launch. The worker sends its first heartbeat when the job arrives. So the first period must cover the launch, the boot, and, for HTTP delivery, the request step, each with its retries. A throttled launch can take several minutes, and the request step retries for up to `retry_window` in each attempt. With a shorter heartbeat timeout, the operation fails with `MicrovmTimeoutError` before the MicroVM sends its first heartbeat. +- The request step needs at least 11 seconds left in the invocation when it starts. It keeps a 10-second reserve before the invocation times out, and it starts a call only with at least 1 second left before the reserve. The launch step can run earlier in the same invocation, so the function timeout must cover it too. A function timeout of `retry_window` plus 11 seconds, 71 seconds by default, leaves the first retry tier about its whole window. With less than 11 seconds left, every attempt fails with `MicrovmEndpointUnavailableError`, and the message of the final `MicrovmDeliveryError` names the function timeout. The Lambda default timeout is 3 seconds. With it, `microvm` runs small jobs in the `run` hook and fails every job that goes over HTTP: a job too large for the hook, and a job with `request.path`. +- The Region comes from the durable execution ARN, or from `AWS_REGION`. +- Without a `client`, the operation creates one `lambda-microvms` client per Region, on first use. It has a 5-second connect timeout, a 30-second read timeout, and 3 attempts in botocore's standard retry mode. So one launch or terminate attempt takes at most about 2 minutes. A call that runs past the function's timeout records no outcome for its step attempt, and the step runs again. The launch is idempotent through its client token, so it is repeated, not doubled. + +### Permissions + +The function's execution role needs: + +- `lambda:RunMicrovm` and `lambda:TerminateMicrovm` on the MicroVM image. Without `lambda:TerminateMicrovm`, every operation still returns its result, but every MicroVM runs until its timeout plus 5 minutes, at most 8 hours. The only sign is an error in the function's log. +- `lambda:CreateMicrovmAuthToken`, for HTTP delivery. Grant it unless every input is small and no config sets a `path`. +- `lambda:PassNetworkConnector` on the connectors that the MicroVM uses. `RunMicrovm` authorizes each connector separately. With the defaults, that is `arn:aws:lambda::aws:network-connector:aws-network-connector:*`. +- `iam:PassRole` on the MicroVM execution role. +- `lambda:GetMicrovm`, for HTTP delivery, from `microvm` and from a session. A job over HTTP checks the MicroVM state after an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED` or a request without an answer. Without `lambda:GetMicrovm`, a job whose request gets that answer, or no answer, fails at once with `MicrovmDeliveryError`, and the message names `lambda:GetMicrovm`. A role without `lambda:GetMicrovm` fails a job whose request gets no answer, even when the MicroVM runs, for example a worker that takes more than 10 seconds to listen. See [The state check](#the-state-check). + +The function's role does not need `lambda:ResumeMicrovm`. The SDK does not resume a MicroVM. The service resumes it on the next request. + +The MicroVM execution role (`execution_role_arn`) needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the function ARN (`arn:aws:lambda:::function::*`). + +## `microvm_session`: run several jobs in one MicroVM + +`microvm_session` launches one MicroVM and runs a handler that sends jobs to it. The jobs share the MicroVM's files and processes. So a later job can use what an earlier job wrote. The MicroVM is terminated when the handler returns or raises. + +```python +import os +from typing import Any + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import DurableContext +from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python_extras.microvm import ( + MicrovmInvokeOptions, + MicrovmSession, + MicrovmSessionConfig, + microvm_session, +) + +SESSION = MicrovmSessionConfig( + image_identifier=os.environ["MICROVM_IMAGE_ARN"], + execution_role_arn=os.environ["MICROVM_ROLE_ARN"], + timeout=Duration.from_hours(1), +) + + +def pipeline(vm: MicrovmSession, ctx: DurableContext) -> Any: + build = vm.invoke( + ctx, + "clone-build", + job_input={"repo": "org/app"}, + options=MicrovmInvokeOptions( + path="/clone-build", + timeout=Duration.from_minutes(15), + heartbeat_timeout=Duration.from_minutes(2), + ), + ) + ctx.wait(Duration.from_minutes(5), name="cool-down") + return vm.invoke( + ctx, + "test", + job_input=build, + options=MicrovmInvokeOptions(timeout=Duration.from_minutes(10)), + ) + + +@durable_execution +def handler(event: dict, context: DurableContext) -> Any: + return microvm_session(context, "pipeline", config=SESSION, handler=pipeline) +``` + +The operation runs its durable operations in one child context named `name`: + +``` +pipeline child context +├── pipeline.session step: generates the session ID +├── pipeline.launch step: RunMicrovm +├── clone-build child context, one per job +│ ├── clone-build.callback callback +│ └── clone-build.request step: POSTs the job +├── cool-down wait +├── test +│ ├── test.callback +│ └── test.request +└── pipeline.terminate step: TerminateMicrovm +``` + +- The client token of the launch is the SHA-256 hex digest of the session ID. The session ID is random, and it is checkpointed. So a retried or replayed launch returns the same MicroVM. +- Every job goes over HTTP. The MicroVM gets the `ALL_INGRESS` connector. A job without a `path` goes to the worker's `handler`. +- The first job is sent right after the launch. The MicroVM endpoint refuses requests until the `run` hook returns, and the request step retries a refused request inside its attempt. So the session needs no separate readiness wait. +- `vm.invoke` takes the context that the call runs in, as `microvm` does. Inside a `map` or `parallel` item, pass the item's context. A job must be created inside the session's child context. A job created in the context that `microvm_session` was called with runs, and the next replay then fails the execution with `NonDeterministicExecutionError`. So `vm.invoke` raises `TypeError` for that context. The core SDK does not check other contexts outside the session. +- `config.timeout` bounds the MicroVM's lifetime: it ends at `timeout` plus 5 minutes, at most 8 hours. Each job waits for its own `timeout`. So set a `heartbeat_timeout` on every job. A job still running when the MicroVM ends then fails at its heartbeat timeout. + +A failed `vm.invoke` raises its MicroVM error inside the handler. The handler can catch it and send another job to the same MicroVM. An error that leaves the handler ends the session: + +1. A MicroVM error that the handler raises itself reaches the caller with its own class, such as `MicrovmJobFailedError`. +2. Any other error reaches the caller as the SDK's `ChildContextError`, as from `run_in_child_context`. This includes a MicroVM error that leaves a child context that the handler created: that child context records `ChildContextError` as its error type. The caller then gets the MicroVM error's message, but not its class. Inside the handler, the `ChildContextError` of that child context has the MicroVM error as its `__cause__`. An SDK error that leaves the handler, such as the `StepError` of a failed step, also reaches the caller as `ChildContextError`, with the SDK error as its `__cause__`. This rule matches `run_in_child_context`. The JavaScript operation differs here: it passes an SDK error to the caller with its own class. + +An invalid job raises `ValidationError`, `TypeError`, or `ValueError` from `vm.invoke`, before the job's durable operations start. The handler can catch it too. A call without the context, `vm.invoke("build", job_input=..., options=...)`, raises Python's `TypeError`. A first argument that is not a durable context, as in `vm.invoke("build", "build", ...)`, raises `AttributeError`. + +The handler's return value is checkpointed as the session's result. So it must be serializable by the SDK, and it counts toward the checkpoint size limit. + +### Suspending an idle MicroVM + +A running MicroVM pays compute charges. A suspended MicroVM pays only for snapshot storage, and it keeps its memory and files. A session can suspend its MicroVM between jobs in two ways: + +1. `auto_suspend_on_idle=True` asks the worker in the MicroVM to suspend its own MicroVM when no job has run for `auto_suspend_idle_time`. The default idle time is 60 seconds, or the session `timeout` if that is shorter. The idle time must be at least 10 seconds. So a session whose `timeout` is under 10 seconds does not suspend by default, even with `auto_suspend_on_idle=True`. +2. `suspend_after=True` on a job makes the durable function suspend the MicroVM after that job. See [Suspending after a job](#suspending-after-a-job). + +To suspend a MicroVM, set `idle_policy` with `"autoResumeEnabled": True`. The SDK does not resume a MicroVM; the service resumes it on the next request: + +```python +config = MicrovmSessionConfig( + image_identifier=IMAGE, + execution_role_arn=ROLE, + timeout=Duration.from_hours(1), + auto_suspend_on_idle=True, + idle_policy={ + "autoResumeEnabled": True, + # Above maximumDurationInSeconds: the 1-hour timeout plus 5 minutes. + "maxIdleDurationSeconds": 4 * 3600, + "suspendedDurationSeconds": 3600, + }, +) +``` + +- Set `maxIdleDurationSeconds` above `maximumDurationInSeconds` (the session `timeout` plus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job. +- Without `autoResumeEnabled`, a job after a suspend fails after its retries with `MicrovmDeliveryError`. With the default retry strategy, that takes about 6 minutes. +- Set `suspendedDurationSeconds` to the session `timeout` or more. What happens to a job when that time ends is not verified. +- A job that arrives more than about 5 seconds after a suspend takes about 1 s longer. A job sent right after a suspend can take up to about 10 seconds, as in the next list. +- The idle policy needs all three fields, and `maxIdleDurationSeconds` must be at least 60. `RunMicrovm` rejects any other policy, and the launch fails with `MicrovmLaunchError`. The session does not check the policy itself. + +A job reaches a suspended MicroVM this way: + +1. For about 5 seconds after a suspend, the endpoint answers a request with HTTP 502 and `x-aws-proxy-error: MICROVM_CONNECT_FAILED`. That request does not resume the MicroVM. The request step retries it, after 250 milliseconds at first, doubling up to 4 seconds. Its 5th retry starts 7.75 seconds after the first request, so the retries reach past that window. With a `retry_window` under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds. +2. After that window, a request resumes the MicroVM. The endpoint holds the request for about 750 milliseconds, and then the worker answers it. +3. After each 502 with `MICROVM_CONNECT_FAILED`, the request step checks the state, at most once per 2 seconds. A `SUSPENDING` or `SUSPENDED` MicroVM keeps the job retrying. See [The state check](#the-state-check). +4. A request that arrives while the MicroVM suspends can get no answer. That request reaches the worker after the next resume. The worker ignores a second request for a callback ID that runs or recently ended. So the job runs once. + +The worker reads its idle time once, from the run hook payload of the launch. `RunMicrovm` sets the idle policy once, at the launch. So a deploy that changes `auto_suspend_on_idle` or `idle_policy` does not change what a running MicroVM does. + +An earlier release resumed a suspended MicroVM itself, and recorded `resumeBeforeDelivery` in the launch checkpoint. An execution in flight that launched with that release replays its launch checkpoint with `resumeBeforeDelivery: true`. This release ignores the field. So the next job of that execution no longer resumes the MicroVM. That job works only if the MicroVM was launched with `autoResumeEnabled: True`. A session that suspended its MicroVM with `suspend_after` and has no `idle_policy` is affected too: its next job after the suspend no longer resumes the MicroVM and fails after its retries with `MicrovmDeliveryError`. + +A MicroVM lives at most its `timeout` plus 5 minutes, and never longer than 8 hours. Every job over HTTP, of a session or of `microvm`, reports a MicroVM that ended: + +- A terminated MicroVM's endpoint answers HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, measured in us-east-1. After that answer, [the state check](#the-state-check) reports the state, and the job fails with `MicrovmNotRunningError`, a subclass of `MicrovmDeliveryError`, after one request. A retry finds the same state, so the job is not retried. The message ends with the request ID of the call that reported the state, for example `GetMicrovm request ID: .`. +- Every session, and every `microvm` job over HTTP, calls `CreateMicrovmAuthToken` before the first request, and again after a 401 or 403. The API model lists `ResourceNotFoundException` for that call. That answer fails the job at once with `MicrovmNotRunningError`, whose message ends with `CreateMicrovmAuthToken request ID: .`. For a terminated MicroVM whose record still exists, the call issues a token, measured in us-east-1. The live service's answer for a removed MicroVM is not verified. + +Suspending is off by default, because it needs more settings and changes what the MicroVM does between jobs: + +- `idle_policy` must set `autoResumeEnabled: True`, as above. +- With `auto_suspend_on_idle`, the MicroVM's role needs `lambda:SuspendMicrovm` on the image. That permission lets code in any MicroVM of the image suspend any other. So use one image per trust boundary. +- Background processes that an earlier job started stop while the MicroVM is suspended. The worker counts only its running jobs. + +The worker must support `auto_suspend_on_idle`. The JavaScript worker [`@aws/durable-execution-sdk-js-microvm-worker`](https://github.com/aws/aws-durable-execution-sdk-js/tree/main/packages/aws-durable-execution-sdk-js-microvm-worker) and the Python worker [`aws-durable-execution-sdk-python-microvm-worker`](../../aws-durable-execution-sdk-python-microvm-worker) both do. Enable the `suspend` and `resume` hooks in the image. The worker then refuses jobs while the MicroVM freezes, and accepts them again as soon as it resumes. Without the `resume` hook, the first job after a resume can wait a little over 30 seconds, until the worker's refusal ends and the request step delivers again. + +### Suspending after a job + +`MicrovmInvokeOptions(suspend_after=True)` makes the durable function suspend the MicroVM after this job. It is an option of one job, not of the session. `microvm` raises `ValidationError` for it, because `microvm` terminates its MicroVM after its one job. A value that is not a `bool` raises `ValidationError` too, before the job's durable operations start. + +```python +build = vm.invoke( + ctx, + "build", + job_input=event, + options=MicrovmInvokeOptions( + timeout=Duration.from_minutes(15), + heartbeat_timeout=Duration.from_seconds(30), + suspend_after=True, + ), +) +approval = ctx.create_callback(name="approval").result() # a long wait +vm.invoke( + ctx, + "deploy", + job_input=build, + options=MicrovmInvokeOptions( + timeout=Duration.from_minutes(10), + heartbeat_timeout=Duration.from_seconds(120), + ), +) +``` + +What happens: + +1. The session calls `SuspendMicrovm` with the session's MicroVM ID when the job's callback result arrives, whether the job succeeded or failed. The call runs inside the job's child context, before the child context returns. A wait for the callback does not suspend the MicroVM. +2. It adds no step and no checkpoint. The child context's own completion checkpoint covers it. On replay, a completed child context does not run again, so the session sends no second suspend. +3. A failed `SuspendMicrovm` call logs a warning and does not fail the job. The MicroVM then keeps running until the idle suspend or the end of the session. +4. The job request carries no field for it, and the worker does nothing for it. The worker needs no change, and the MicroVM's role needs no suspend permission. +5. The next job's first request gets HTTP 502, and the service resumes the MicroVM on a later request. So the session needs the `idle_policy` of [Suspending an idle MicroVM](#suspending-an-idle-microvm), with `autoResumeEnabled: True`. + +The setting works with or without `auto_suspend_on_idle`. With both, the function suspends right after the job, and the worker's idle time applies after each resume. + +The call is not a durable operation, so these rules are yours to follow. The session adds no guard for them: + +1. Do not use it on a job that runs next to other jobs of the session (`map`, `parallel`, promise combinators). Those jobs freeze until a later request makes the service resume the MicroVM, and can hit their heartbeat timeout. +2. The suspend runs inside the job's child context, so it follows the child context replay rules. A result larger than the child context checkpoint limit makes the child context run again on replay, and each run suspends again. +3. A retry after the Lambda invocation fails can send the suspend again. The invocation can also end while the call runs, when other operations wait. The call is then sent again. Suspending a suspended MicroVM succeeds. +4. Adding or removing `suspend_after` changes what the session does on replay. Change it only through a new function version. + +Use `suspend_after` only on the job before a long wait, such as a durable wait or a callback: + +- Not on back-to-back jobs. Each suspend writes a snapshot, and each resume adds about 1 second before the next job, or up to about 10 seconds when the job is sent right after the suspend. +- Not on `map` items or other jobs that run at the same time. The suspend freezes the jobs that still run. +- Not on the last job. The session terminates the MicroVM after its handler returns, so the snapshot is wasted. + +It needs: + +1. The `resume` hook in the image. Without it, the worker refuses the next job with HTTP 503 for up to 30 seconds after the suspend. +2. A `heartbeat_timeout` on the next job of at least its `retry_window` plus 30 seconds. The heartbeat timeout starts before delivery, so it includes the resume and the refusal. +3. An `idle_policy` with `autoResumeEnabled: True`, as in [Suspending an idle MicroVM](#suspending-an-idle-microvm). +4. `lambda:SuspendMicrovm` on the image in the function's role. The MicroVM's role needs `lambda:SuspendMicrovm` only for `auto_suspend_on_idle`. + +The suspend runs after a job timeout or a heartbeat timeout too, so a job that still runs in the MicroVM is frozen, not stopped. + +The call ends at the invocation reserve, like the other calls of the session. A failed suspend, or a call that does not end in time, is not retried. The MicroVM may then still be running until the idle suspend or the end of the session. The warning names `lambda:SuspendMicrovm` for a missing permission. + +### The state check + +Every job over HTTP, of a session or of `microvm`, checks the MicroVM state after a sign that the MicroVM does not run. The check finds a MicroVM that ended. It does not resume a MicroVM. + +1. The sign is an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, or a request without an answer. A booting MicroVM's endpoint holds the request until the worker listens. A boot gives no such sign, unless the worker takes more than 10 seconds to start listening. Then the first delivery gets no answer, and the operation makes one GetMicrovm call. +2. The request step calls `GetMicrovm` after each such answer, at most once per 2 seconds. The call runs first, before the `retry_window` and the backoff are checked, and only the invocation reserve limits it. So a delayed answer that uses up the window still gets the check. With less than 1 second left before the reserve, no call starts, and the attempt ends with `MicrovmEndpointUnavailableError`, which the default retry strategy retries. +3. `TERMINATING`, `TERMINATED`, or not found fails the job at once with `MicrovmNotRunningError`. +4. `RUNNING`, `PENDING`, `SUSPENDING`, or `SUSPENDED` keeps the job retrying. The service resumes a suspended MicroVM on a request, when its idle policy sets `autoResumeEnabled`. +5. A denied `GetMicrovm` fails the job at once with `MicrovmDeliveryError`. A retry is denied the same way, so the default retry strategy does not retry it. The message names `lambda:GetMicrovm` and ends with the `GetMicrovm` request ID. +6. Any other failed `GetMicrovm`, such as throttling, a 5xx answer, a timeout, or a connection error, logs one warning per step attempt, so up to 5 with the default retry strategy: `GetMicrovm failed, so the operation could not check whether the MicroVM has ended. The request step keeps sending the job.` The request step then keeps sending the job. + +### Session permissions + +A session needs the permissions of `microvm` with HTTP delivery: `lambda:RunMicrovm`, `lambda:TerminateMicrovm`, `lambda:CreateMicrovmAuthToken`, `lambda:PassNetworkConnector`, and `iam:PassRole`. Suspending adds the permissions listed above: `lambda:SuspendMicrovm` in the function's role for `suspend_after`, and in the MicroVM's role for `auto_suspend_on_idle`. Grant `lambda:GetMicrovm` to every session's function role, as for `microvm` with HTTP delivery. The role does not need `lambda:ResumeMicrovm`. `lambda:GetMicrovm` is required. Without it, a job whose request gets an HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, or no answer, fails at once, with a message that names `lambda:GetMicrovm`. The check runs after any such answer, at most once per 2 seconds. + +### Subtypes + +The JavaScript operations record MicroVM subtypes, such as `Microvm` on the child context, `MicrovmSession` on a session, and `MicrovmLaunch` on the launch step. The core SDK accepts a custom `sub_type` on `StepConfig`, `ChildConfig`, and `CallbackConfig`. This package does not set it yet, so it records the SDK's default subtypes. + +On replay, the SDK compares the subtype of each operation with its checkpoint. A mismatch fails the execution with `NonDeterministicExecutionError`. So a release that changed the subtypes would fail every execution that waits on a MicroVM during the deploy. So this package must record the MicroVM subtypes from its first published release. + +## Best practices for MicroVM operations + +The MicroVM operations handle the network, the service, and the worker lifecycle. They do not check every setting. This section lists what your code and your deploy must do. Each rule states what happens if you break it. + +### Deploys and versions + +1. Invoke the durable function through a published version or an alias, not `$LATEST`. + - An execution started through a version or an alias is pinned to that version. Every replay runs the same code and the same configuration, including the environment variables. + - An execution started through `$LATEST` is not pinned. After a deploy, it resumes on the new code. See [Invoking durable Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking.html). + - So under `$LATEST`, a launch step that retries after a deploy can fail with `MicrovmLaunchError`. A MicroVM that an earlier attempt of that step created can then keep running, and be billed, until its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. +2. Do not read MicroVM settings from a source that can change during an execution, such as an SSM parameter, AppConfig, or an S3 object. Code outside a step runs again on every replay. So it reads the value again. A value that changes between two attempts of a launch step can fail the launch, as in rule 1. Read such a value inside a step. A replay then uses the checkpointed value. +3. The MicroVM packages are experimental. A release can change the durable operations that the MicroVM operations record. [Subtypes](#subtypes) describes one such change. An execution that replays its history on such a release fails with `NonDeterministicExecutionError`. So let executions that started on an earlier build end on that build. Under rule 1, this needs no extra step: a new version starts only new executions, and running executions continue on their own version. Under `$LATEST`, let the running executions end before you deploy a new build. + +### Function timeout + +1. Set the function timeout so that each request step starts with at least 11 seconds left in the invocation. With less time left, the step attempt fails at once. A function timeout under 11 seconds fails every attempt. So every job over HTTP fails with `MicrovmDeliveryError`, and its message says to raise the function timeout. Every session job goes over HTTP. [Defaults and limits](#defaults-and-limits) explains the 11 seconds and gives the recommended timeout. + +### Job inputs + +1. Pass a JSON-serializable input with finite numbers. Any other input raises `TypeError` before any durable operation starts. See [Errors](#errors). +2. Keep the job input's JSON at or under `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, in UTF-8. Pass larger data by reference, for example as an S3 key. + - The Python worker reads a job request body of at most 6 MiB. The body holds the input's JSON and a few job fields. + - `microvm` and `vm.invoke` measure the input's JSON before any durable operation starts. A larger input raises `ValueError`. So `microvm` launches no MicroVM for it. +3. Expect a job to go over HTTP in three cases: + - The `run` hook payload of a `microvm` job is over 4,096 code points. The payload holds the input, the callback ID, and a few other fields. + - The `microvm` config sets `request.path`. The job then goes over HTTP for any input size. + - The job is a session job. Every session job goes over HTTP. + + A job over HTTP needs four things that a `run` hook job does not: + - `lambda:CreateMicrovmAuthToken` in the function's role. Without it, the job fails with `MicrovmDeliveryError`. See [Permissions](#permissions). + - The function timeout of [Function timeout](#function-timeout). With a shorter timeout, the job fails with `MicrovmDeliveryError`. + - Ingress to the MicroVM. The operation gives such a MicroVM the `ALL_INGRESS` connector. If you set `ingress_network_connectors`, include a connector that allows ingress. Without ingress, the request step cannot reach the MicroVM's endpoint. So the job fails. + - `lambda:PassNetworkConnector` on the `ALL_INGRESS` connector in the function's role. Its ARN is `arn:aws:lambda::aws:network-connector:aws-network-connector:ALL_INGRESS`. `RunMicrovm` authorizes each connector separately. So a role that allows `NO_INGRESS` and `INTERNET_EGRESS`, but not `ALL_INGRESS`, fails every HTTP launch with `MicrovmLaunchError`. + + A `microvm` job whose payload fits the `run` hook, and whose config has no `request.path`, needs none of these four. So a function that passed its tests with such jobs can fail every job over HTTP. See [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). + +### Job handlers + +1. Follow [Best practices for job handlers](../../aws-durable-execution-sdk-python-microvm-worker/docs/worker.md#best-practices-for-job-handlers) in the worker README for the code that runs each job in the MicroVM. +2. Make the side effects of a job safe to repeat. + - The worker ignores a second delivery of a callback ID. It remembers the callback IDs of its last 1,000 finished jobs, and it forgets them when its process restarts. + - Each `microvm` call and each `vm.invoke` creates a new callback ID. So when your code runs a job again after a failure, the worker runs it again, also in the same MicroVM. + - A failure can follow a run that already changed something. After `MicrovmLaunchError` with `run` hook delivery, the MicroVM can still run the job. An HTTP launch request contains no job. The request step starts only after the launch succeeds. So when you invoke through a version or an alias, no MicroVM runs the job after `MicrovmLaunchError` with HTTP delivery. After `MicrovmTimeoutError`, the job can have done part of its work. See [Errors](#errors). + +### The HTTP job route + +1. Answer the job request itself with a 2xx status at once, and run the job in the background. The worker package does this for its routes. + - The request step waits at most 10 seconds for an answer, and then sends the job again. So a route that answers only when a longer job ends receives the job again and again. The delivery then fails with `MicrovmDeliveryError`, after the retries of both tiers. + - The operation does not follow redirects. It fails the job at once with `MicrovmDeliveryError` for a 3xx answer, and for a 4xx answer other than 401, 403, and 429. A proxy in front of the route that redirects, for example to add a trailing slash, causes the same failure. See [Retries in two tiers](#retries-in-two-tiers). +2. Ignore a second request for a callback ID that the route already ran. The same job can arrive more than once. A route served without the worker package must do this itself, or it runs the job again. See [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). +3. Give `path` a value that follows the path rules in [How the job reaches the MicroVM](#how-the-job-reaches-the-microvm). The same check rejects a query, a fragment, and a path that starts with `/aws/lambda-microvms/`, where Lambda sends the lifecycle hooks. An invalid path raises `ValidationError` when `microvm` or `vm.invoke` is called, before its durable operations start. Register the same string as the worker's route. The worker answers a path that it does not serve with HTTP 404, and the operation then fails the job at once with `MicrovmDeliveryError`. + +### Sessions + +1. Call `vm.invoke(ctx, name, job_input=..., options=...)` with the context that the session handler receives, or a context nested in it. A job created in another context runs, and the next replay fails the execution with `NonDeterministicExecutionError`. `vm.invoke` raises `TypeError` only for the context that `microvm_session` was called with. It does not detect any other wrong context. See [`microvm_session`](#microvm_session-run-several-jobs-in-one-microvm). +2. Pass a function as `handler`. `microvm_session` does not check it before it starts. A value that is not callable raises Python's `TypeError` when the session calls it, after the launch. The session then terminates the MicroVM, and the caller receives `ChildContextError`. So the mistake costs one MicroVM boot. mypy reports it before that. +3. If you set `auto_suspend_idle_time`, meet three conditions: + - `auto_suspend_on_idle` is `True`. + - The idle time is at least 10 seconds. + - The idle time is at most the session `timeout`. + + A value that breaks any one condition raises `ValidationError` when `microvm_session` is called, before any durable operation starts. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). +4. Set `suspend_after` only on the job before a long wait. Never set it on a job that runs next to other jobs of the session. Add `lambda:SuspendMicrovm` to the function's role, set an `idle_policy` with `autoResumeEnabled: True`, and change `suspend_after` only through a new function version. Give the next job a `heartbeat_timeout` of at least its `retry_window` plus 30 seconds. On back-to-back jobs, each suspend and resume adds latency and a snapshot. A shorter heartbeat timeout can fail the next job with `MicrovmTimeoutError` while the MicroVM resumes. See [Suspending after a job](#suspending-after-a-job). +5. A session that suspends its MicroVM, with `auto_suspend_on_idle` or `suspend_after`, needs an `idle_policy` with `autoResumeEnabled: True` and a `maxIdleDurationSeconds` above the session `timeout` plus 5 minutes. Without `autoResumeEnabled`, every job after a suspend fails with `MicrovmDeliveryError`. A shorter `maxIdleDurationSeconds` suspends a running job. Change `auto_suspend_on_idle` or `idle_policy` only through a new version, as in [Deploys and versions](#deploys-and-versions). The launch sets both, so a deploy changes them only for sessions that launch after it. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). + +### Retry strategies + +1. Keep `default_microvm_retry_strategy` unless you need another policy. It decides by the error class, the error code, and the HTTP status. A strategy that retries less fails a job on a throttle or a network error. A strategy that retries more repeats errors that a retry cannot change, such as `AccessDeniedException`, and the job fails later. See [Retries in two tiers](#retries-in-two-tiers). +2. In a custom `retry_strategy`, match an AWS error by its code, `error.response["Error"]["Code"]`. Match an error of the operation by its class, such as `MicrovmEndpointUnavailableError`. The API model names each error code. It defines each message only as a string. So the service can change a message without an API change. A strategy that matches the message then decides differently, and nothing reports the change. + +### Custom clients and tests + +1. A `client` that you pass must behave as a boto3 `lambda-microvms` client. + - It returns outputs in the boto3 shape, with `ResponseMetadata`. A `RunMicrovm` output without `microvmId`, or without `endpoint` for HTTP delivery, fails the launch with `MicrovmLaunchError`. An output without `ResponseMetadata.RequestId` gives error messages without a request ID. + - It raises botocore `ClientError` with a `response` that holds `Error.Code` and `ResponseMetadata.HTTPStatusCode`. The default strategy reads both. It fails the step at once for an error of any other class, unless the error is a timeout or a connection failure. + - It sets connect and read timeouts, such as the 5 and 30 seconds of the default client. botocore's own defaults are 60 seconds each. A `RunMicrovm` or `TerminateMicrovm` call that runs past the function timeout records no outcome for its step attempt. So the step runs again. A `CreateMicrovmAuthToken` or `GetMicrovm` call that reaches the invocation reserve fails its attempt. Its thread keeps running until the call returns. So the call can still reach the service after the step stopped waiting for it. +2. Pass an `opener` when the network reaches the MicroVM endpoint only through a proxy. The default opener ignores the proxy settings in the environment. So without an `opener`, the launch succeeds, and every POST fails. The job then fails with `MicrovmDeliveryError`, after the retries of both tiers. See [Retries in two tiers](#retries-in-two-tiers). +3. In tests, wrap a real `lambda-microvms` client in botocore's `Stubber`. `Stubber` checks each stubbed output against the API model. Its `add_client_error` raises `ClientError` with `Error.Code` and `ResponseMetadata.HTTPStatusCode`. So the operation and the default strategy read the same fields as from the service. A fake with another shape tests paths that the service never produces. Add `ResponseMetadata` with a `RequestId` to a stubbed output, or pass `response_meta` to `add_client_error`, to test the request IDs in messages. moto has no `lambda-microvms` backend, as of moto 5.2.4. + +### Permissions and monitoring + +1. Give the function's execution role every permission in [Permissions](#permissions), including `lambda:TerminateMicrovm`. Without it, every operation still returns its result, and every MicroVM runs, and is billed, until its maximum duration. +2. Create an alarm on the log line that starts with `MicroVM "": terminate failed for `, for example with a CloudWatch Logs metric filter that matches `terminate failed for`. A failed terminate does not fail the operation. The operation logs this error instead. The MicroVM then keeps running, and is billed, until its maximum duration: the operation timeout plus 5 minutes, at most 8 hours. This line is the only sign. See [What it does](#what-it-does). diff --git a/packages/aws-durable-execution-sdk-python-microvm-worker/README.md b/packages/aws-durable-execution-sdk-python-microvm-worker/README.md index a693d578a..c4e159fef 100644 --- a/packages/aws-durable-execution-sdk-python-microvm-worker/README.md +++ b/packages/aws-durable-execution-sdk-python-microvm-worker/README.md @@ -2,227 +2,169 @@ > This package is experimental. Its API can change in any release. -This package runs inside an AWS Lambda MicroVM. It receives jobs from a durable function, sends heartbeats while each job runs, and reports each job's result or error to the durable function's callback. +Run jobs from a durable function inside an AWS Lambda MicroVM, and report each job's result to the durable function. -The package does not depend on the durable execution SDK, because it runs in the MicroVM and not in the durable function. The job document is the same in every SDK language. So a durable function in any language can send jobs to this worker. +The worker runs in the MicroVM image, not in the durable function. It does not depend on the durable execution SDK. The job document is the same in every SDK language. So a durable function in any language can send jobs to this worker. -## Usage +## Install -```python -import logging +```console +pip install aws-durable-execution-sdk-python-microvm-worker +``` -from aws_durable_execution_sdk_python_microvm_worker import start_microvm_worker +## Quick start +```python +import logging -def run_build(job_input, context): - # context.cancelled is set when the durable function stops waiting. - return {"passed": build(job_input["repo"], stop=context.cancelled)} +from aws_durable_execution_sdk_python_microvm_worker import ( + MicrovmJobContext, + start_microvm_worker, +) -def clone_build(job_input, context): - return clone_and_build(job_input) +def run_build(job_input: dict, context: MicrovmJobContext) -> dict: + return {"repo": job_input["repo"], "passed": True} if __name__ == "__main__": logging.basicConfig(level=logging.INFO) - worker = start_microvm_worker( - # Jobs that name no route. A small job arrives in the run hook, and a - # large one over HTTP. The handler receives both the same way. - handler=run_build, - # Jobs sent to a named route. - routes={"/clone-build": clone_build}, - ) + worker = start_microvm_worker(handler=run_build) worker.wait() ``` -The worker needs `handler`, `routes`, or both. A handler is a plain function. An `async def` function, or an object whose `__call__` is `async def`, raises `TypeError`. A plain function that returns a coroutine fails its job with `ResultSerializationError`, because a coroutine is not JSON-serializable. Python then warns that the coroutine was never awaited. The handler's return value becomes the durable function's result, and must be JSON-serializable. An exception fails the durable operation with the exception's class name and message. `context.cancelled` is set when the durable function stops waiting, for example after a heartbeat timeout, and when Lambda calls the `terminate` hook. Python cannot stop a running thread, so the handler should check the event and stop its work. - -`start_microvm_worker` serves in a daemon thread and returns. So the entry point ends with `worker.wait()`, which blocks until `worker.close()`. - -## Image configuration - -Start the worker when the image's process starts. Lambda snapshots the process after the `ready` hook, so the worker must already be listening. - -The image needs these hook settings in `CreateMicrovmImage`: - -```json -{ - "hooks": { - "port": 8080, - "microvmHooks": { - "run": "ENABLED", - "runTimeoutInSeconds": 10, - "resume": "ENABLED", - "resumeTimeoutInSeconds": 10, - "suspend": "ENABLED", - "suspendTimeoutInSeconds": 10, - "terminate": "ENABLED", - "terminateTimeoutInSeconds": 10 - }, - "microvmImageHooks": { "ready": "ENABLED" } - } -} -``` - -- The `run` hook delivers the job, or tells the worker that jobs arrive over HTTP. So it must be enabled. -- The `resume` hook tells a session's worker that its MicroVM runs again after a suspend. It is optional, and recommended for sessions that suspend. A session that suspends after a job, or when idle, needs it. See [Suspending when idle](#suspending-when-idle) and [Suspending after a job](#suspending-after-a-job). -- The `suspend` hook tells the worker that Lambda is about to freeze the MicroVM. The worker then refuses new jobs until the `resume` hook. It is optional, and recommended for sessions. -- The `terminate` hook tells the worker that Lambda is about to end the MicroVM. The worker then fails the callback of each running job, so the durable function fails the job at once instead of at its heartbeat timeout or its timeout. It is optional, and recommended. Give it at least 10 seconds, because the worker reports before it answers, for at most 5 seconds. A hook timeout left unset ended the hook after about 2 seconds in testing. -- The service requires the `ready` hook whenever a lifecycle hook is enabled. The worker answers it with HTTP 200. -- `port` must match the worker's port. The worker listens on 8080 by default, which is also the port that the MicroVM endpoint forwards to. Pass `port=` for another port. - -The worker listens on IPv6 and IPv4 at once when the host supports it. Lambda calls the hooks on `localhost`, and that name can resolve to either family. - -## What the worker does - -On `POST /aws/lambda-microvms/runtime/v1/run` the worker: - -1. Validates the run hook payload. -2. Answers HTTP 200 at once, because a slow or failed `run` hook can terminate the MicroVM. -3. Runs `handler` for the payload's job, when it has one. A payload without a job belongs to a MicroVM whose jobs arrive over HTTP. - -The worker acts only on the first valid `run` hook. The MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token for the MicroVM. So a later `run` hook gets HTTP 200, and is ignored. - -On `POST ` the worker validates the job request, answers HTTP 202 at once, and runs the route's handler. `POST /durable-execution/v1/job` (`MICROVM_JOB_PATH`) runs `handler`. That path is reserved, so a route cannot use it. A job request can arrive before the `run` hook. Until the `run` hook arrives, the worker takes the MicroVM ID for `context.microvm_id` from the first job request that has a `microvmId`. The `run` hook's ID then replaces it for later jobs. Without either, `context.microvm_id` is `"unknown"`. +Start the worker when the image's process starts. `start_microvm_worker` serves in a daemon thread and returns. So the entry point ends with `worker.wait()`. -Each job runs in its own thread. For every job, the worker: +The handler's return value becomes the durable function's result. An exception fails the job with the exception's class name and message. -1. Sends heartbeats while the job runs and while its outcome is reported, when the job has a heartbeat timeout. See [Heartbeats](#heartbeats). -2. Sets `context.cancelled` when a heartbeat finds the callback gone while the handler runs. Python cannot stop a running thread, so the handler should check the event and stop its work. The worker then does not report the outcome. -3. Completes the callback with the handler's result, or fails it with the handler's exception. A `SystemExit` from the handler is reported as a failure too. -4. Reports a result that is over 256 KB, or that is not JSON-serializable, as a failure. So the durable function learns why at once, instead of at its timeout. +## Common uses -A second delivery of a job is answered like the first and ignored, while the job runs and after it ended. The durable function can deliver the same job again: it retries a delivery whose answer it did not receive, and its request step runs again on replay when its result was not checkpointed. The worker remembers the callback IDs of the last 1,000 finished jobs. When a job ends and the worker already holds 1,000 IDs, it removes the oldest one. So a job runs once, unless at least 1,000 other jobs ended in between, or the worker process restarted. While the worker refuses new jobs after a suspend or after the `terminate` hook, a second delivery gets one of three answers: +### Serve named routes -1. A second delivery of a finished job gets HTTP 202 in every state. It starts nothing. A 503 would make the durable function deliver the job again only to get this answer. -2. A second delivery of a known job gets HTTP 202 after the `terminate` hook. That hook fails a running job's callback. So the durable function learns the outcome from the callback. -3. A second delivery of a running job gets HTTP 503 during a suspend refusal, as a new job does. The running job can freeze with the MicroVM. A 202 would end the delivery, and nothing would resume the MicroVM. The job would then wait for its heartbeat timeout or its timeout. The 503 makes the durable function deliver the job again. The service resumes the MicroVM on that request. That delivery gets HTTP 202, and the job completes once. - -The worker creates a Lambda client for each job, after the job arrives, not at image build. A client created before the snapshot would carry build-time state into every MicroVM. A client kept across jobs would keep connections that a suspend of the MicroVM can leave dead. The worker closes each job's default client when the job ends. It never closes a client from `create_client`. - -A document that does not match the contract gets HTTP 400 with the validation message. If the document named a callback, the worker also fails that callback, so the durable function fails at once instead of at its timeout. A request whose body cannot be read gets HTTP 400 with a generic message, and the error is logged. The worker accepts a `run` hook body of at most 64 KiB and a job request body of at most 6 MiB, with a `Content-Length` or with chunked transfer encoding. In a chunked body, the chunk-size lines, their extensions, and the trailers have a limit of their own, of the same size. A larger body gets HTTP 400. The worker sends it without reading the body, and then closes the connection. The unread bytes make the kernel send a TCP reset in place of a normal close. So a client that is still sending can get the reset before it reads the 400. A request with `Expect: 100-continue` first gets `100 Continue`, as `http.server` sends it. The client then sends the body, and gets the 400 or the reset in the same way. The MicroVM operations of both SDK extras packages, JS (`@aws/durable-execution-sdk-js-extras`) and Python (`aws-durable-execution-sdk-python-extras`), reject a job input that does not fit before they send the job. So only another sender can get the reset. Other lifecycle hooks get HTTP 200. Other paths get HTTP 404. - -`worker.idle(timeout)` waits until every job started so far has ended, and a running `SuspendMicrovm` call has returned. It returns `False` when the timeout ends first. `worker.close()` stops the server and the idle time. Running jobs continue, so call `idle()` first to let them report their outcome. - -## Suspending when idle +```python +from aws_durable_execution_sdk_python_microvm_worker import ( + MicrovmJobContext, + start_microvm_worker, +) -A session's run hook payload can set `autoSuspendIdleSeconds`. The session sets it only with `auto_suspend_on_idle=True`, because suspending is off by default. The SDK does not resume a MicroVM. So the session must also set an `idle_policy` with `autoResumeEnabled: True`, and the service resumes the MicroVM on the next request. The worker then suspends its own MicroVM when it has run no job for that many seconds: -1. The idle time starts when the `run` hook arrives without a job, when a job ends, and when the `resume` lifecycle hook arrives, if the image enables it. -2. A job that starts cancels the idle time. So the worker never suspends the MicroVM during a job. -3. When the idle time ends, the worker calls `SuspendMicrovm` with the MicroVM ID from the `run` hook, never one from a job request. The call returned in under 100 milliseconds in testing, before the MicroVM froze. The default client makes one attempt. An attempt can suspend the MicroVM and lose its answer in the freeze. A retry would then run after the MicroVM resumed. The `resume` hook has ended the refusal of jobs by then. So the retry could suspend the MicroVM with the next job inside. With one attempt, a 5xx answer or a dropped connection on the first attempt of the default client counts as unknown, as in item 8. botocore's retry would often have succeeded. So the worker refuses jobs for up to 30 seconds on a MicroVM that did not suspend. No `resume` hook ends that refusal, because nothing suspended. This delay is the accepted cost of one attempt. No attempt of the call starts after 10 seconds, because a `before-send` handler on the client stops botocore's retries at that deadline. The same handler starts no attempt after a `resume` hook. This covers a client from `create_microvms_client` that retries. botocore cannot cancel an attempt in flight, and it opens the connection after `before-send`. So an attempt that started before the deadline can still reach the service after it. The worker waits for that attempt to end, and refuses jobs until then. The client's connect and read timeouts bound the attempt: 5 and 10 seconds for the default client. A client from `create_microvms_client` should use similar timeouts. botocore's own defaults are 60 seconds each, so with them the refusal can last minutes. The service resumes the MicroVM on the next job's request, when the MicroVM's idle policy sets `autoResumeEnabled`, and the process continues with its memory and files. -4. From the moment the idle time ends, the worker answers every new job with HTTP 503 and does not start it. A job accepted then would freeze with the MicroVM, and nothing would resume it. After a 503, the durable function delivers the job again. A request to the suspended MicroVM makes the service resume it. -5. The refusal ends when the `resume` lifecycle hook arrives, or 30 seconds after the call returned, if no hook arrives. The idle time then starts again. At most one call runs at a time. A `resume` hook that arrives during the call ends the refusal at once, and the next idle time starts when the call ends. The call then starts no further attempt, and the worker logs `stopping the suspend call: the MicroVM resumed during it` at info level. A second `resume` hook during the same call logs nothing. The JavaScript worker cancels the attempt in flight. This worker can only stop the next attempt, because botocore cannot cancel an attempt in flight. So enable the `resume` hook in the image. Without it, the first job after a resume can wait a little over 30 seconds: the 30 seconds, plus up to 4 seconds until the durable function's next delivery attempt. A test measured 32 seconds. -6. A call that the service rejected on its first attempt, such as `AccessDeniedException` for a missing permission, is logged as a warning, and the MicroVM keeps running. Missing credentials count as a rejection. A later call in the same MicroVM most likely finds no credentials either. The worker accepts jobs again at once and does not retry the call. The next job that ends starts the idle time again. The exception is a `resume` hook that arrived during the call. The idle time then starts when the call ends. So the worker retries the call one idle time after the call ends, if no job runs and no `suspend` hook refusal is in effect when the call ends. A second exception is a `suspend` hook that arrived during the call and whose refusal is still in effect when the call ends. The MicroVM is then suspending anyway. So the rejection does not end the hook's refusal. That refusal ends on the `resume` hook, or 30 seconds after the last `suspend` hook. The idle time then starts again. -7. A call that the service refused for now on its first attempt is logged as a warning too, and the worker accepts jobs again at once. This covers expired credentials and a signature error, with or without a clock offset, whatever the HTTP status. The service answers these before it acts. botocore retries none of these errors, so they reach the worker on the first attempt. The idle time then starts again, so the worker tries the call again. That idle time lasts at least 30 seconds, also when `autoSuspendIdleSeconds` is shorter. The same error can repeat on every call, for example with a wrong secret key or an expired session token from environment variables. So while no job ends and no lifecycle hook arrives, the worker makes at most one call per 30 seconds, and logs one warning per call. A job that ends after the refusal, or a `resume` hook after it, starts the normal idle time instead. A `resume` hook or a job that ends during the call does not shorten the idle time after the refusal, because the refused outcome arrives after it. A `suspend` hook whose refusal is still in effect when the call ends is the exception, as in item 6. Its refusal stays, and the idle time starts when it ends. The default client makes one attempt. So a throttle from it counts as refused for now. A client from `create_microvms_client` that retries a throttle turns it into an unknown outcome, as in item 8. The JavaScript worker classifies these errors the same way. -8. A call whose outcome is unknown is handled like a successful call, because the service may have suspended the MicroVM anyway. This covers a timeout, a conflict, a server error, and a dropped connection. An error after botocore retried an earlier attempt also counts as unknown, because the earlier attempt may have suspended the MicroVM. Only a client from `create_microvms_client` that retries can raise such an error. A call that the `before-send` handler stopped after a `resume` hook counts as unknown too. The handler can stop the first attempt, and then nothing was sent. An earlier attempt, if there was one, may have suspended the MicroVM. That `resume` hook has already ended the refusal. So the worker accepts jobs at once, and starts no new 30-second limit for that call. The refusal continues until the `resume` hook or the 30-second limit. The idle time then starts again, so the worker tries the call again. A `resume` hook during the call, or the end of a `suspend` hook's 30 seconds during the call, has already ended the refusal. The worker then accepts jobs at once, and tries the call again one idle time after the call and every job end. -9. The worker logs one warning for each failed call. The message names the outcome. The fields are, in order, `microvmId`, `error`, `outcome`, `suspendHookArrived`, `terminating`, `closed`, and `refusingJobs`. `microvmId` is the MicroVM ID of the call. `error` holds the name and the message of the error. `outcome` is `rejected`, `refusedForNow`, or `unknown`. `suspendHookArrived` states whether a `suspend` hook arrived since the worker decided to make the call. `terminating` states whether a `terminate` hook came before the worker acted on the outcome. `closed` states whether `worker.close()` came before the worker acted on the outcome. `refusingJobs` states whether the worker answers jobs with HTTP 503 after it has acted on the outcome. The worker also logs two info lines about the call. It logs `suspending the idle MicroVM` before each call, with `microvmId`, `idleSeconds`, and `waitedSeconds`. It logs `stopping the suspend call: the MicroVM resumed during it` for the first `resume` hook during a call, with `microvmId`. The JavaScript worker logs the same stop line. +def run_build(job_input: dict, context: MicrovmJobContext) -> dict: + return {"passed": True} -`worker.close()` stops the idle time. A closed worker never suspends its MicroVM again. A refusal that is already running still ends on the `resume` hook or after 30 seconds. -The worker counts only jobs. It cannot see background threads or processes, or inbound traffic that is not a job. A session with such work leaves `auto_suspend_on_idle` off, and the payload then has no `autoSuspendIdleSeconds`. +def clone_build(job_input: dict, context: MicrovmJobContext) -> dict: + return {"cloned": job_input["repo"]} -The worker creates a `lambda-microvms` client for each suspend, and closes it after the call. Pass `create_microvms_client` to provide the client. The worker never closes a client that it returns. Give the client one attempt, as the default client has. The worker starts no attempt after a `resume` hook. A retry that starts before the `resume` hook arrives can still suspend the MicroVM after it resumed. A client that retries must report its retries in `ResponseMetadata.RetryAttempts` of the error that it raises, as botocore does. The worker otherwise counts a retried call as a first attempt. An earlier attempt of that call may have suspended the MicroVM. So a retried throttle would count as refused for now, and the worker would accept a job that then freezes with the MicroVM. -## Suspending after a job +if __name__ == "__main__": + worker = start_microvm_worker( + handler=run_build, # Jobs that name no route. + routes={"/clone-build": clone_build}, # Jobs sent to this path. + ) + worker.wait() +``` -The durable function suspends a session's MicroVM after a job. `vm.invoke` with `suspend_after=True` calls `SuspendMicrovm` itself, from the durable function, after the job's result arrives. The job request carries no field for it, and the worker does nothing for it. +The durable function sends a job to a route with `MicrovmRequestConfig(path=...)` or `MicrovmInvokeOptions(path=...)`. Use the same string in both places. -So the MicroVM's execution role needs no `lambda:SuspendMicrovm` for this case. The durable function's role needs it. See the [extras README](../aws-durable-execution-sdk-python-extras/README.md#suspending-after-a-job). +### Stop a cancelled job -Enable the `suspend` and `resume` hooks in the image. The worker answers the `suspend` hook, refuses jobs while the MicroVM freezes, and accepts them again on the `resume` hook. Without the `resume` hook, the worker refuses the next job with HTTP 503 until 30 seconds after the `suspend` hook. The next job's heartbeat timeout counts that time. +`context.cancelled` is a `threading.Event`. The worker sets it when the durable function stops waiting, for example after a heartbeat timeout, and when Lambda calls the `terminate` hook. Python cannot stop a running thread. So check the event in long work. -## Suspend and terminate hooks +```python +import subprocess -Lambda waits for the answer to a `suspend` or `terminate` hook, up to the hook's timeout. The network and the execution role's credentials work while the hook runs. These facts come from probes in us-east-1, which the JavaScript SDK ran. +from aws_durable_execution_sdk_python_microvm_worker import MicrovmJobContext -On the `suspend` hook the worker: -1. Answers HTTP 200 at once. A suspend hook that ran past its timeout terminated the MicroVM instead of suspending it. -2. Refuses every new job, and every second delivery of a running job, with HTTP 503, as after its own suspend. A job accepted now would freeze with the MicroVM. The refusal ends when the `resume` hook arrives, or after 30 seconds. -3. Leaves running jobs alone. They freeze with the MicroVM, and continue after a resume. -4. Defers a job from a `run` hook whose body was still arriving. The durable function cannot deliver a `run` hook job again, so the worker does not refuse it. It starts the job when the refusal ends. +def run_build(job_input: dict, context: MicrovmJobContext) -> dict: + for command in job_input["commands"]: + if context.cancelled.is_set(): + return {"cancelled": True} + subprocess.run(command, shell=False, check=True) + return {"passed": True} +``` -Lambda called the hook after `SuspendMicrovm` from outside, after the worker's own `SuspendMicrovm`, and when the idle policy suspended the MicroVM. +### Run async code -On the `terminate` hook the worker: +A handler must be a plain function. An `async def` handler raises `TypeError`. Call `asyncio.run` inside the handler instead: -1. Sets each running job's `context.cancelled`, and stops its heartbeats. -2. Fails each running job's callback with `MicrovmTerminatedError`. The worker reports `MicrovmTerminatedError` as the callback's error type. The `microvm` operation then fails with a `MicrovmJobFailedError` whose message contains `(MicrovmTerminatedError)`. The worker does not report the handler's outcome when it returns later. -3. Skips a job whose callback a heartbeat already found gone. A report would fail the same way. -4. Answers HTTP 200 when the reports end, or after 5 seconds. A report took 20 to 1,700 milliseconds in testing. The answer also waits for a job that is reporting its own outcome, for a `run` hook request whose body is still arriving, and for every callback failure report in flight, such as the report for an invalid request. Work that begins during the wait is included. A second `terminate` hook that arrives during the wait waits for the same work. Lambda can end the MicroVM as soon as it gets the answer, so a report sent after it can be lost. -5. Refuses every later new job with HTTP 503, and never suspends the MicroVM again. A second delivery of a known job gets HTTP 202. That job has ended, or this hook fails its callback. So the durable function learns the outcome from the callback, and a 503 would only make it deliver the job again. A job whose callback a heartbeat already found gone gets HTTP 202 too. That callback is gone, so the durable function no longer waits for it. A job from a `run` hook whose body was still arriving is not started. Its callback fails with `MicrovmTerminatedError`, as does a job deferred by an earlier `suspend` hook. +```python +import asyncio -Lambda called the hook after `TerminateMicrovm` on a running MicroVM, and at the end of `maximumDurationInSeconds`. It did not call the hook when it terminated a suspended MicroVM, after `TerminateMicrovm` or at the end of the idle policy's `suspendedDurationSeconds`. The process is frozen then. A job of such a MicroVM fails at its heartbeat timeout or its timeout. +from aws_durable_execution_sdk_python_microvm_worker import MicrovmJobContext -The MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token for the MicroVM. Lambda's own hook calls carried `Host: localhost:8080` and no `x-amzn-requestid` header. A forwarded request carried the endpoint's host name and an `x-amzn-requestid` header. So the worker acts on `suspend`, `terminate`, and `resume` only when the request has a `localhost`, `127.0.0.1`, or `[::1]` host, with any port, and no `x-amzn-requestid`. It answers any other request to these hooks with 200, and logs a warning. The Lambda documentation does not state these headers. If Lambda changed them, the worker would ignore `suspend` and `terminate`. A refusal after a suspend would then end after 30 seconds instead of at the `resume` hook. Without the check on `resume`, a forwarded `resume` between a `suspend` hook and the freeze could end the refusal early, and a job accepted then would freeze with the MicroVM. The `run` hook does not use the check, because a false rejection there would lose every job. -## Lower-level building blocks +async def build(job_input: dict) -> dict: + await asyncio.sleep(1) + return {"passed": True} -- `MicrovmWorkerListener.create(...)` takes the same arguments as `start_microvm_worker`, except `port`. Call its `close()` when you stop serving it, so that it no longer suspends the MicroVM. Its `request_handler_class` serves the hooks and routes from your own `http.server.ThreadingHTTPServer`. Set the server's backlog to 511, as `start_microvm_worker` does. A session can send many jobs at once, and the default backlog of 5 resets the connections beyond it. The server's constructor already listens, so set `request_queue_size = 511` as a class attribute of a `ThreadingHTTPServer` subclass, not on the server object. -- `RunHookRequest.from_dict` and `MicrovmJobRequest.from_dict` validate the two documents that deliver a job: the `run` lifecycle hook body, and the body of an HTTP job request. -- `CallbackReporter` sends heartbeats, and completes the callback with a result or an error. -## The job documents +def run_build(job_input: dict, context: MicrovmJobContext) -> dict: + return asyncio.run(build(job_input)) +``` -The `run` hook body that Lambda sends: +## Advanced -```json -{ "microvmId": "", "runHookPayload": "" } -``` +### Image configuration -The `runHookPayload` string holds: +Set these hooks in `CreateMicrovmImage`. `port` must match the worker's port, 8080 by default. ```json { - "version": 1, - "region": "us-east-1", - "job": { "callbackId": "", "heartbeatTimeoutSeconds": 60, "input": {} } + "hooks": { + "port": 8080, + "microvmHooks": { + "run": "ENABLED", + "runTimeoutInSeconds": 10, + "resume": "ENABLED", + "resumeTimeoutInSeconds": 10, + "suspend": "ENABLED", + "suspendTimeoutInSeconds": 10, + "terminate": "ENABLED", + "terminateTimeoutInSeconds": 10 + }, + "microvmImageHooks": { "ready": "ENABLED" } + } } ``` -`job` is absent when the job arrives over HTTP. An HTTP job request body has the job fields at the top level, with `version`, `region`, and an optional `microvmId`. `MicrovmJobRequest` holds those job fields in its `job` attribute. - -A document that does not match raises `InvalidRunHookPayloadError`. When the document named a non-empty callback ID, the error carries `callback_id` and `region`, so the worker can fail that callback at once. - -## Reporting +- The `run` hook is required. It delivers small jobs, or tells the worker that jobs arrive over HTTP. +- The `suspend` and `resume` hooks are needed for sessions that suspend. Without `resume`, the first job after a resume can wait about 30 seconds. +- The `terminate` hook lets the worker fail running jobs at once when the MicroVM ends. Give it at least 10 seconds. +- The service requires the `ready` hook when any lifecycle hook is enabled. -`CallbackReporter.succeed` sends a JSON result of at most 256 KB. A larger result raises `ResultTooLargeError`, and a result that is not JSON-serializable raises `ResultSerializationError`. In both cases no call is made, so the caller can report a failure instead. A `float` that is `nan` or infinite is not JSON, so it raises `ResultSerializationError`. +### Suspending -`CallbackReporter.fail` sends the error's class name and message, cut to 256 and 8,192 characters. The traceback is not sent, because it would expose the image's file paths in the durable execution history. +The worker suspends its own MicroVM when the session sets `auto_suspend_on_idle=True`. It never suspends during a job. The SDK never calls `ResumeMicrovm`. So the session's MicroVM must set `idlePolicy.autoResumeEnabled`, and the service resumes it on the next request. Setting that is the job of the durable function's developer. Set `maxIdleDurationSeconds` above the MicroVM's `maximumDurationInSeconds`. -Each completion makes up to 5 attempts, and each attempt ends after 30 seconds. A permanent error, such as `AccessDeniedException` or a closed callback, is not retried. The service answers `CallbackTimeoutException` for a callback that is already complete, timed out, or whose execution stopped. That answer after an attempt with an unknown outcome means that an earlier attempt most likely delivered the outcome. The reporter then logs a warning and returns. +The worker counts only jobs. It does not see background threads, processes, or other inbound traffic. Leave `auto_suspend_on_idle` off for a session with such work. -## Heartbeats +With `suspend_after`, the durable function suspends the MicroVM itself. The worker does nothing for it. -When the job sets `heartbeatTimeoutSeconds`, `Heartbeats` sends a heartbeat at once, and then about every third of the heartbeat timeout, at most every 15 minutes. Each wait is 1 to 2 seconds shorter than the interval, so MicroVMs that start together send at different moments. The jitter comes from a hash of the callback ID, not from the `random` module. Lambda restores every MicroVM from one snapshot of the worker process, so `random` would return the same values in each of them. +### Results and heartbeats -Each heartbeat call ends after a third of the interval. After a failed heartbeat, the next one comes after an eighth to a quarter of the interval, for at most two failures in a row. With these limits, two failed or stalled calls in a row still stay within the heartbeat timeout: +- Return a JSON-serializable value of at most 256 KB. The worker reports any other value as a failure of the job. Store a larger result elsewhere, for example in S3, and return its key. +- The worker sends heartbeats while a job runs, when the job sets a heartbeat timeout. +- The worker ignores a second delivery of a job that it already ran. It remembers the last 1,000 finished jobs. So make side effects safe to repeat across process restarts. -1. The service times the heartbeat timeout from when it receives a heartbeat. It can receive a call at the call's start or at its end. -2. So the worst gap runs from a good call received at its start, through two failed calls and their retry waits, to a good call received at its end. -3. With interval I, that gap is at most about 2.83 I. The heartbeat timeout is at least 3 I. +### Logging -A third failure in a row waits a full interval, and the job can then reach its heartbeat timeout. The first heartbeat of a job also resolves credentials and opens a connection within its call timeout. With the default interval, that is a ninth of the heartbeat timeout. So a heartbeat timeout under about 15 seconds can lose the first heartbeat. The quick retry usually covers it. +The worker logs to the standard library logger `aws_durable_execution_sdk_python_microvm_worker`. Pass `logger=` to replace it. Configure `logging` in the image to see its INFO lines. -An explicit `heartbeat_interval_seconds` must be above 0 and at most 900, and it is cut to a third of the job's heartbeat timeout. Other values raise `ValueError`. - -## Logging - -The worker logs to the standard library logger `aws_durable_execution_sdk_python_microvm_worker`, and passes its structured fields in `extra`. Every line about a job carries its `callbackId`. `CallbackReporter.create` and the heartbeats take the same `logger` argument. Configure `logging` in the image to see its INFO lines, for example with `logging.basicConfig(level=logging.INFO)`. Any object with the `logging.Logger` signatures for `info`, `warning`, and `error` can replace it. A logger that raises does not stop the worker: the line is dropped. - -## Permissions +### Permissions The MicroVM's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the durable function's ARN (`arn:aws:lambda:::function::*`). -A session that suspends when idle also needs `lambda:SuspendMicrovm` on the MicroVM image (`arn:aws:lambda:::microvm-image:`). The action authorizes on the image, not on the MicroVM. So a MicroVM with this permission can suspend any MicroVM from the same image, even during a job. That job then waits until its heartbeat timeout or its timeout. Use one image per trust boundary. +With `auto_suspend_on_idle`, the role also needs `lambda:SuspendMicrovm` on the MicroVM image. That permission lets a MicroVM suspend any MicroVM of the same image. So use one image per trust boundary. + +The worker needs boto3 1.43.35 or later. -The worker needs boto3 1.43.35 or later, the first release with the `lambda-microvms` client. +## Reference -## Best practices for job handlers +- [docs/worker.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-microvm-worker/docs/worker.md): the request flow, the lifecycle hooks, idle suspend outcomes, the job documents, reporting, heartbeats, logging fields, and best practices. +- [extras](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-extras): the `microvm` and `microvm_session` operations that send jobs. +- [Source](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker): lower-level building blocks such as `MicrovmWorkerListener` and `CallbackReporter`. -These rules apply to the `handler` and the route handlers that you pass to `start_microvm_worker`. Each rule states what happens if a handler breaks it. +## License -1. Write each handler as a plain function. To run async code, call `asyncio.run` inside the handler. The worker runs each job in a thread of its own, and that thread has no running event loop. So `asyncio.run` can start one there. An `async def` handler makes `start_microvm_worker` raise `TypeError`, and a handler that returns a coroutine fails each of its jobs with `ResultSerializationError`. See [Usage](#usage). -2. Return a JSON-serializable value of at most 256 KB. The worker reports any other value as a failure of the job. So the durable function raises `MicrovmJobFailedError`. Store a larger result elsewhere, for example in S3, and return its key. See [Reporting](#reporting). -3. Pass a `logger` that does not raise. The worker drops each line whose logger raises, and it continues. So that line is lost, and nothing reports the loss. See [Logging](#logging). +This project is licensed under the [Apache-2.0 License](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/LICENSE). diff --git a/packages/aws-durable-execution-sdk-python-microvm-worker/docs/worker.md b/packages/aws-durable-execution-sdk-python-microvm-worker/docs/worker.md new file mode 100644 index 000000000..375b99bbe --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-microvm-worker/docs/worker.md @@ -0,0 +1,196 @@ +# MicroVM worker: reference + +This file describes in detail what the worker does. +The [README](../README.md) shows how to use it. + +## Handler rules + +The worker needs `handler`, `routes`, or both. A handler is a plain function. An `async def` function, or an object whose `__call__` is `async def`, raises `TypeError`. A plain function that returns a coroutine fails its job with `ResultSerializationError`, because a coroutine is not JSON-serializable. Python then warns that the coroutine was never awaited. The handler's return value becomes the durable function's result, and must be JSON-serializable. An exception fails the durable operation with the exception's class name and message. `context.cancelled` is set when the durable function stops waiting, for example after a heartbeat timeout, and when Lambda calls the `terminate` hook. Python cannot stop a running thread, so the handler should check the event and stop its work. + +## Image configuration + +Start the worker when the image's process starts. Lambda snapshots the process after the `ready` hook, so the worker must already be listening. + +The image needs these hook settings in `CreateMicrovmImage`: + +```json +{ + "hooks": { + "port": 8080, + "microvmHooks": { + "run": "ENABLED", + "runTimeoutInSeconds": 10, + "resume": "ENABLED", + "resumeTimeoutInSeconds": 10, + "suspend": "ENABLED", + "suspendTimeoutInSeconds": 10, + "terminate": "ENABLED", + "terminateTimeoutInSeconds": 10 + }, + "microvmImageHooks": { "ready": "ENABLED" } + } +} +``` + +- The `run` hook delivers the job, or tells the worker that jobs arrive over HTTP. So it must be enabled. +- The `resume` hook tells a session's worker that its MicroVM runs again after a suspend. It is optional, and recommended for sessions that suspend. A session that suspends after a job, or when idle, needs it. See [Suspending when idle](#suspending-when-idle) and [Suspending after a job](#suspending-after-a-job). +- The `suspend` hook tells the worker that Lambda is about to freeze the MicroVM. The worker then refuses new jobs until the `resume` hook. It is optional, and recommended for sessions. +- The `terminate` hook tells the worker that Lambda is about to end the MicroVM. The worker then fails the callback of each running job, so the durable function fails the job at once instead of at its heartbeat timeout or its timeout. It is optional, and recommended. Give it at least 10 seconds, because the worker reports before it answers, for at most 5 seconds. A hook timeout left unset ended the hook after about 2 seconds in testing. +- The service requires the `ready` hook whenever a lifecycle hook is enabled. The worker answers it with HTTP 200. +- `port` must match the worker's port. The worker listens on 8080 by default, which is also the port that the MicroVM endpoint forwards to. Pass `port=` for another port. + +The worker listens on IPv6 and IPv4 at once when the host supports it. Lambda calls the hooks on `localhost`, and that name can resolve to either family. + +## What the worker does + +On `POST /aws/lambda-microvms/runtime/v1/run` the worker: + +1. Validates the run hook payload. +2. Answers HTTP 200 at once, because a slow or failed `run` hook can terminate the MicroVM. +3. Runs `handler` for the payload's job, when it has one. A payload without a job belongs to a MicroVM whose jobs arrive over HTTP. + +The worker acts only on the first valid `run` hook. The MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token for the MicroVM. So a later `run` hook gets HTTP 200, and is ignored. + +On `POST ` the worker validates the job request, answers HTTP 202 at once, and runs the route's handler. `POST /durable-execution/v1/job` (`MICROVM_JOB_PATH`) runs `handler`. That path is reserved, so a route cannot use it. A job request can arrive before the `run` hook. Until the `run` hook arrives, the worker takes the MicroVM ID for `context.microvm_id` from the first job request that has a `microvmId`. The `run` hook's ID then replaces it for later jobs. Without either, `context.microvm_id` is `"unknown"`. + +Each job runs in its own thread. For every job, the worker: + +1. Sends heartbeats while the job runs and while its outcome is reported, when the job has a heartbeat timeout. See [Heartbeats](#heartbeats). +2. Sets `context.cancelled` when a heartbeat finds the callback gone while the handler runs. Python cannot stop a running thread, so the handler should check the event and stop its work. The worker then does not report the outcome. +3. Completes the callback with the handler's result, or fails it with the handler's exception. A `SystemExit` from the handler is reported as a failure too. +4. Reports a result that is over 256 KB, or that is not JSON-serializable, as a failure. So the durable function learns why at once, instead of at its timeout. + +A second delivery of a job is answered like the first and ignored, while the job runs and after it ended. The durable function can deliver the same job again: it retries a delivery whose answer it did not receive, and its request step runs again on replay when its result was not checkpointed. The worker remembers the callback IDs of the last 1,000 finished jobs. When a job ends and the worker already holds 1,000 IDs, it removes the oldest one. So a job runs once, unless at least 1,000 other jobs ended in between, or the worker process restarted. While the worker refuses new jobs after a suspend or after the `terminate` hook, a second delivery gets one of three answers: + +1. A second delivery of a finished job gets HTTP 202 in every state. It starts nothing. A 503 would make the durable function deliver the job again only to get this answer. +2. A second delivery of a known job gets HTTP 202 after the `terminate` hook. That hook fails a running job's callback. So the durable function learns the outcome from the callback. +3. A second delivery of a running job gets HTTP 503 during a suspend refusal, as a new job does. The running job can freeze with the MicroVM. A 202 would end the delivery, and nothing would resume the MicroVM. The job would then wait for its heartbeat timeout or its timeout. The 503 makes the durable function deliver the job again. The service resumes the MicroVM on that request. That delivery gets HTTP 202, and the job completes once. + +The worker creates a Lambda client for each job, after the job arrives, not at image build. A client created before the snapshot would carry build-time state into every MicroVM. A client kept across jobs would keep connections that a suspend of the MicroVM can leave dead. The worker closes each job's default client when the job ends. It never closes a client from `create_client`. + +A document that does not match the contract gets HTTP 400 with the validation message. If the document named a callback, the worker also fails that callback, so the durable function fails at once instead of at its timeout. A request whose body cannot be read gets HTTP 400 with a generic message, and the error is logged. The worker accepts a `run` hook body of at most 64 KiB and a job request body of at most 6 MiB, with a `Content-Length` or with chunked transfer encoding. In a chunked body, the chunk-size lines, their extensions, and the trailers have a limit of their own, of the same size. A larger body gets HTTP 400. The worker sends it without reading the body, and then closes the connection. The unread bytes make the kernel send a TCP reset in place of a normal close. So a client that is still sending can get the reset before it reads the 400. A request with `Expect: 100-continue` first gets `100 Continue`, as `http.server` sends it. The client then sends the body, and gets the 400 or the reset in the same way. The MicroVM operations of both SDK extras packages, JS (`@aws/durable-execution-sdk-js-extras`) and Python (`aws-durable-execution-sdk-python-extras`), reject a job input that does not fit before they send the job. So only another sender can get the reset. Other lifecycle hooks get HTTP 200. Other paths get HTTP 404. + +`worker.idle(timeout)` waits until every job started so far has ended, and a running `SuspendMicrovm` call has returned. It returns `False` when the timeout ends first. `worker.close()` stops the server and the idle time. Running jobs continue, so call `idle()` first to let them report their outcome. + +## Suspending when idle + +A session's run hook payload can set `autoSuspendIdleSeconds`. The session sets it only with `auto_suspend_on_idle=True`, because suspending is off by default. The SDK does not resume a MicroVM. So the session must also set an `idle_policy` with `autoResumeEnabled: True`, and the service resumes the MicroVM on the next request. The worker then suspends its own MicroVM when it has run no job for that many seconds: + +1. The idle time starts when the `run` hook arrives without a job, when a job ends, and when the `resume` lifecycle hook arrives, if the image enables it. +2. A job that starts cancels the idle time. So the worker never suspends the MicroVM during a job. +3. When the idle time ends, the worker calls `SuspendMicrovm` with the MicroVM ID from the `run` hook, never one from a job request. The call returned in under 100 milliseconds in testing, before the MicroVM froze. The default client makes one attempt. An attempt can suspend the MicroVM and lose its answer in the freeze. A retry would then run after the MicroVM resumed. The `resume` hook has ended the refusal of jobs by then. So the retry could suspend the MicroVM with the next job inside. With one attempt, a 5xx answer or a dropped connection on the first attempt of the default client counts as unknown, as in item 8. botocore's retry would often have succeeded. So the worker refuses jobs for up to 30 seconds on a MicroVM that did not suspend. No `resume` hook ends that refusal, because nothing suspended. This delay is the accepted cost of one attempt. No attempt of the call starts after 10 seconds, because a `before-send` handler on the client stops botocore's retries at that deadline. The same handler starts no attempt after a `resume` hook. This covers a client from `create_microvms_client` that retries. botocore cannot cancel an attempt in flight, and it opens the connection after `before-send`. So an attempt that started before the deadline can still reach the service after it. The worker waits for that attempt to end, and refuses jobs until then. The client's connect and read timeouts bound the attempt: 5 and 10 seconds for the default client. A client from `create_microvms_client` should use similar timeouts. botocore's own defaults are 60 seconds each, so with them the refusal can last minutes. The service resumes the MicroVM on the next job's request, when the MicroVM's idle policy sets `autoResumeEnabled`, and the process continues with its memory and files. +4. From the moment the idle time ends, the worker answers every new job with HTTP 503 and does not start it. A job accepted then would freeze with the MicroVM, and nothing would resume it. After a 503, the durable function delivers the job again. A request to the suspended MicroVM makes the service resume it. +5. The refusal ends when the `resume` lifecycle hook arrives, or 30 seconds after the call returned, if no hook arrives. The idle time then starts again. At most one call runs at a time. A `resume` hook that arrives during the call ends the refusal at once, and the next idle time starts when the call ends. The call then starts no further attempt, and the worker logs `stopping the suspend call: the MicroVM resumed during it` at info level. A second `resume` hook during the same call logs nothing. The JavaScript worker cancels the attempt in flight. This worker can only stop the next attempt, because botocore cannot cancel an attempt in flight. So enable the `resume` hook in the image. Without it, the first job after a resume can wait a little over 30 seconds: the 30 seconds, plus up to 4 seconds until the durable function's next delivery attempt. A test measured 32 seconds. +6. A call that the service rejected on its first attempt, such as `AccessDeniedException` for a missing permission, is logged as a warning, and the MicroVM keeps running. Missing credentials count as a rejection. A later call in the same MicroVM most likely finds no credentials either. The worker accepts jobs again at once and does not retry the call. The next job that ends starts the idle time again. The exception is a `resume` hook that arrived during the call. The idle time then starts when the call ends. So the worker retries the call one idle time after the call ends, if no job runs and no `suspend` hook refusal is in effect when the call ends. A second exception is a `suspend` hook that arrived during the call and whose refusal is still in effect when the call ends. The MicroVM is then suspending anyway. So the rejection does not end the hook's refusal. That refusal ends on the `resume` hook, or 30 seconds after the last `suspend` hook. The idle time then starts again. +7. A call that the service refused for now on its first attempt is logged as a warning too, and the worker accepts jobs again at once. This covers expired credentials and a signature error, with or without a clock offset, whatever the HTTP status. The service answers these before it acts. botocore retries none of these errors, so they reach the worker on the first attempt. The idle time then starts again, so the worker tries the call again. That idle time lasts at least 30 seconds, also when `autoSuspendIdleSeconds` is shorter. The same error can repeat on every call, for example with a wrong secret key or an expired session token from environment variables. So while no job ends and no lifecycle hook arrives, the worker makes at most one call per 30 seconds, and logs one warning per call. A job that ends after the refusal, or a `resume` hook after it, starts the normal idle time instead. A `resume` hook or a job that ends during the call does not shorten the idle time after the refusal, because the refused outcome arrives after it. A `suspend` hook whose refusal is still in effect when the call ends is the exception, as in item 6. Its refusal stays, and the idle time starts when it ends. The default client makes one attempt. So a throttle from it counts as refused for now. A client from `create_microvms_client` that retries a throttle turns it into an unknown outcome, as in item 8. The JavaScript worker classifies these errors the same way. +8. A call whose outcome is unknown is handled like a successful call, because the service may have suspended the MicroVM anyway. This covers a timeout, a conflict, a server error, and a dropped connection. An error after botocore retried an earlier attempt also counts as unknown, because the earlier attempt may have suspended the MicroVM. Only a client from `create_microvms_client` that retries can raise such an error. A call that the `before-send` handler stopped after a `resume` hook counts as unknown too. The handler can stop the first attempt, and then nothing was sent. An earlier attempt, if there was one, may have suspended the MicroVM. That `resume` hook has already ended the refusal. So the worker accepts jobs at once, and starts no new 30-second limit for that call. The refusal continues until the `resume` hook or the 30-second limit. The idle time then starts again, so the worker tries the call again. A `resume` hook during the call, or the end of a `suspend` hook's 30 seconds during the call, has already ended the refusal. The worker then accepts jobs at once, and tries the call again one idle time after the call and every job end. +9. The worker logs one warning for each failed call. The message names the outcome. The fields are, in order, `microvmId`, `error`, `outcome`, `suspendHookArrived`, `terminating`, `closed`, and `refusingJobs`. `microvmId` is the MicroVM ID of the call. `error` holds the name and the message of the error. `outcome` is `rejected`, `refusedForNow`, or `unknown`. `suspendHookArrived` states whether a `suspend` hook arrived since the worker decided to make the call. `terminating` states whether a `terminate` hook came before the worker acted on the outcome. `closed` states whether `worker.close()` came before the worker acted on the outcome. `refusingJobs` states whether the worker answers jobs with HTTP 503 after it has acted on the outcome. The worker also logs two info lines about the call. It logs `suspending the idle MicroVM` before each call, with `microvmId`, `idleSeconds`, and `waitedSeconds`. It logs `stopping the suspend call: the MicroVM resumed during it` for the first `resume` hook during a call, with `microvmId`. The JavaScript worker logs the same stop line. + +`worker.close()` stops the idle time. A closed worker never suspends its MicroVM again. A refusal that is already running still ends on the `resume` hook or after 30 seconds. + +The worker counts only jobs. It cannot see background threads or processes, or inbound traffic that is not a job. A session with such work leaves `auto_suspend_on_idle` off, and the payload then has no `autoSuspendIdleSeconds`. + +The worker creates a `lambda-microvms` client for each suspend, and closes it after the call. Pass `create_microvms_client` to provide the client. The worker never closes a client that it returns. Give the client one attempt, as the default client has. The worker starts no attempt after a `resume` hook. A retry that starts before the `resume` hook arrives can still suspend the MicroVM after it resumed. A client that retries must report its retries in `ResponseMetadata.RetryAttempts` of the error that it raises, as botocore does. The worker otherwise counts a retried call as a first attempt. An earlier attempt of that call may have suspended the MicroVM. So a retried throttle would count as refused for now, and the worker would accept a job that then freezes with the MicroVM. + +## Suspending after a job + +The durable function suspends a session's MicroVM after a job. `vm.invoke` with `suspend_after=True` calls `SuspendMicrovm` itself, from the durable function, after the job's result arrives. The job request carries no field for it, and the worker does nothing for it. + +So the MicroVM's execution role needs no `lambda:SuspendMicrovm` for this case. The durable function's role needs it. See the [extras README](../../aws-durable-execution-sdk-python-extras/docs/microvm.md#suspending-after-a-job). + +Enable the `suspend` and `resume` hooks in the image. The worker answers the `suspend` hook, refuses jobs while the MicroVM freezes, and accepts them again on the `resume` hook. Without the `resume` hook, the worker refuses the next job with HTTP 503 until 30 seconds after the `suspend` hook. The next job's heartbeat timeout counts that time. + +## Suspend and terminate hooks + +Lambda waits for the answer to a `suspend` or `terminate` hook, up to the hook's timeout. The network and the execution role's credentials work while the hook runs. These facts come from probes in us-east-1, which the JavaScript SDK ran. + +On the `suspend` hook the worker: + +1. Answers HTTP 200 at once. A suspend hook that ran past its timeout terminated the MicroVM instead of suspending it. +2. Refuses every new job, and every second delivery of a running job, with HTTP 503, as after its own suspend. A job accepted now would freeze with the MicroVM. The refusal ends when the `resume` hook arrives, or after 30 seconds. +3. Leaves running jobs alone. They freeze with the MicroVM, and continue after a resume. +4. Defers a job from a `run` hook whose body was still arriving. The durable function cannot deliver a `run` hook job again, so the worker does not refuse it. It starts the job when the refusal ends. + +Lambda called the hook after `SuspendMicrovm` from outside, after the worker's own `SuspendMicrovm`, and when the idle policy suspended the MicroVM. + +On the `terminate` hook the worker: + +1. Sets each running job's `context.cancelled`, and stops its heartbeats. +2. Fails each running job's callback with `MicrovmTerminatedError`. The worker reports `MicrovmTerminatedError` as the callback's error type. The `microvm` operation then fails with a `MicrovmJobFailedError` whose message contains `(MicrovmTerminatedError)`. The worker does not report the handler's outcome when it returns later. +3. Skips a job whose callback a heartbeat already found gone. A report would fail the same way. +4. Answers HTTP 200 when the reports end, or after 5 seconds. A report took 20 to 1,700 milliseconds in testing. The answer also waits for a job that is reporting its own outcome, for a `run` hook request whose body is still arriving, and for every callback failure report in flight, such as the report for an invalid request. Work that begins during the wait is included. A second `terminate` hook that arrives during the wait waits for the same work. Lambda can end the MicroVM as soon as it gets the answer, so a report sent after it can be lost. +5. Refuses every later new job with HTTP 503, and never suspends the MicroVM again. A second delivery of a known job gets HTTP 202. That job has ended, or this hook fails its callback. So the durable function learns the outcome from the callback, and a 503 would only make it deliver the job again. A job whose callback a heartbeat already found gone gets HTTP 202 too. That callback is gone, so the durable function no longer waits for it. A job from a `run` hook whose body was still arriving is not started. Its callback fails with `MicrovmTerminatedError`, as does a job deferred by an earlier `suspend` hook. + +Lambda called the hook after `TerminateMicrovm` on a running MicroVM, and at the end of `maximumDurationInSeconds`. It did not call the hook when it terminated a suspended MicroVM, after `TerminateMicrovm` or at the end of the idle policy's `suspendedDurationSeconds`. The process is frozen then. A job of such a MicroVM fails at its heartbeat timeout or its timeout. + +The MicroVM endpoint forwards a request to a hook path from any caller that holds an auth token for the MicroVM. Lambda's own hook calls carried `Host: localhost:8080` and no `x-amzn-requestid` header. A forwarded request carried the endpoint's host name and an `x-amzn-requestid` header. So the worker acts on `suspend`, `terminate`, and `resume` only when the request has a `localhost`, `127.0.0.1`, or `[::1]` host, with any port, and no `x-amzn-requestid`. It answers any other request to these hooks with 200, and logs a warning. The Lambda documentation does not state these headers. If Lambda changed them, the worker would ignore `suspend` and `terminate`. A refusal after a suspend would then end after 30 seconds instead of at the `resume` hook. Without the check on `resume`, a forwarded `resume` between a `suspend` hook and the freeze could end the refusal early, and a job accepted then would freeze with the MicroVM. The `run` hook does not use the check, because a false rejection there would lose every job. + +## Lower-level building blocks + +- `MicrovmWorkerListener.create(...)` takes the same arguments as `start_microvm_worker`, except `port`. Call its `close()` when you stop serving it, so that it no longer suspends the MicroVM. Its `request_handler_class` serves the hooks and routes from your own `http.server.ThreadingHTTPServer`. Set the server's backlog to 511, as `start_microvm_worker` does. A session can send many jobs at once, and the default backlog of 5 resets the connections beyond it. The server's constructor already listens, so set `request_queue_size = 511` as a class attribute of a `ThreadingHTTPServer` subclass, not on the server object. +- `RunHookRequest.from_dict` and `MicrovmJobRequest.from_dict` validate the two documents that deliver a job: the `run` lifecycle hook body, and the body of an HTTP job request. +- `CallbackReporter` sends heartbeats, and completes the callback with a result or an error. + +## The job documents + +The `run` hook body that Lambda sends: + +```json +{ "microvmId": "", "runHookPayload": "" } +``` + +The `runHookPayload` string holds: + +```json +{ + "version": 1, + "region": "us-east-1", + "job": { "callbackId": "", "heartbeatTimeoutSeconds": 60, "input": {} } +} +``` + +`job` is absent when the job arrives over HTTP. An HTTP job request body has the job fields at the top level, with `version`, `region`, and an optional `microvmId`. `MicrovmJobRequest` holds those job fields in its `job` attribute. + +A document that does not match raises `InvalidRunHookPayloadError`. When the document named a non-empty callback ID, the error carries `callback_id` and `region`, so the worker can fail that callback at once. + +## Reporting + +`CallbackReporter.succeed` sends a JSON result of at most 256 KB. A larger result raises `ResultTooLargeError`, and a result that is not JSON-serializable raises `ResultSerializationError`. In both cases no call is made, so the caller can report a failure instead. A `float` that is `nan` or infinite is not JSON, so it raises `ResultSerializationError`. + +`CallbackReporter.fail` sends the error's class name and message, cut to 256 and 8,192 characters. The traceback is not sent, because it would expose the image's file paths in the durable execution history. + +Each completion makes up to 5 attempts, and each attempt ends after 30 seconds. A permanent error, such as `AccessDeniedException` or a closed callback, is not retried. The service answers `CallbackTimeoutException` for a callback that is already complete, timed out, or whose execution stopped. That answer after an attempt with an unknown outcome means that an earlier attempt most likely delivered the outcome. The reporter then logs a warning and returns. + +## Heartbeats + +When the job sets `heartbeatTimeoutSeconds`, `Heartbeats` sends a heartbeat at once, and then about every third of the heartbeat timeout, at most every 15 minutes. Each wait is 1 to 2 seconds shorter than the interval, so MicroVMs that start together send at different moments. The jitter comes from a hash of the callback ID, not from the `random` module. Lambda restores every MicroVM from one snapshot of the worker process, so `random` would return the same values in each of them. + +Each heartbeat call ends after a third of the interval. After a failed heartbeat, the next one comes after an eighth to a quarter of the interval, for at most two failures in a row. With these limits, two failed or stalled calls in a row still stay within the heartbeat timeout: + +1. The service times the heartbeat timeout from when it receives a heartbeat. It can receive a call at the call's start or at its end. +2. So the worst gap runs from a good call received at its start, through two failed calls and their retry waits, to a good call received at its end. +3. With interval I, that gap is at most about 2.83 I. The heartbeat timeout is at least 3 I. + +A third failure in a row waits a full interval, and the job can then reach its heartbeat timeout. The first heartbeat of a job also resolves credentials and opens a connection within its call timeout. With the default interval, that is a ninth of the heartbeat timeout. So a heartbeat timeout under about 15 seconds can lose the first heartbeat. The quick retry usually covers it. + +An explicit `heartbeat_interval_seconds` must be above 0 and at most 900, and it is cut to a third of the job's heartbeat timeout. Other values raise `ValueError`. + +## Logging + +The worker logs to the standard library logger `aws_durable_execution_sdk_python_microvm_worker`, and passes its structured fields in `extra`. Every line about a job carries its `callbackId`. `CallbackReporter.create` and the heartbeats take the same `logger` argument. Configure `logging` in the image to see its INFO lines, for example with `logging.basicConfig(level=logging.INFO)`. Any object with the `logging.Logger` signatures for `info`, `warning`, and `error` can replace it. A logger that raises does not stop the worker: the line is dropped. + +## Permissions + +The MicroVM's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the durable function's ARN (`arn:aws:lambda:::function::*`). + +A session that suspends when idle also needs `lambda:SuspendMicrovm` on the MicroVM image (`arn:aws:lambda:::microvm-image:`). The action authorizes on the image, not on the MicroVM. So a MicroVM with this permission can suspend any MicroVM from the same image, even during a job. That job then waits until its heartbeat timeout or its timeout. Use one image per trust boundary. + +The worker needs boto3 1.43.35 or later, the first release with the `lambda-microvms` client. + +## Best practices for job handlers + +These rules apply to the `handler` and the route handlers that you pass to `start_microvm_worker`. Each rule states what happens if a handler breaks it. + +1. Write each handler as a plain function. To run async code, call `asyncio.run` inside the handler. The worker runs each job in a thread of its own, and that thread has no running event loop. So `asyncio.run` can start one there. An `async def` handler makes `start_microvm_worker` raise `TypeError`, and a handler that returns a coroutine fails each of its jobs with `ResultSerializationError`. See [Usage](#handler-rules). +2. Return a JSON-serializable value of at most 256 KB. The worker reports any other value as a failure of the job. So the durable function raises `MicrovmJobFailedError`. Store a larger result elsewhere, for example in S3, and return its key. See [Reporting](#reporting). +3. Pass a `logger` that does not raise. The worker drops each line whose logger raises, and it continues. So that line is lost, and nothing reports the loss. See [Logging](#logging). diff --git a/packages/aws-durable-execution-sdk-python-testing/README.md b/packages/aws-durable-execution-sdk-python-testing/README.md index cae654f2d..b32e5111c 100644 --- a/packages/aws-durable-execution-sdk-python-testing/README.md +++ b/packages/aws-durable-execution-sdk-python-testing/README.md @@ -2,145 +2,106 @@ [![PyPI - Version](https://img.shields.io/pypi/v/aws-durable-execution-sdk-python-testing.svg)](https://pypi.org/project/aws-durable-execution-sdk-python-testing) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/aws-durable-execution-sdk-python-testing.svg)](https://pypi.org/project/aws-durable-execution-sdk-python-testing) - - [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/aws/aws-durable-execution-sdk-python-testing/badge)](https://scorecard.dev/viewer/?uri=github.com/aws/aws-durable-execution-sdk-python-testing) ------ - -## Table of Contents +Test Python durable functions locally, before you deploy them. -- [Installation](#installation) -- [Quick Start](#quick-start) -- [Testing functions that invoke other functions](#testing-functions-that-invoke-other-functions) -- [Architecture](#architecture) -- [Documentation](#documentation) -- [Developer Guide](#developers) -- [License](#license) - -## Installation +## Install ```console pip install aws-durable-execution-sdk-python-testing ``` -## Overview - -Use the AWS Durable Execution Testing SDK for Python to test your Python durable functions locally. - -The test framework contains a local runner, so you can run and test your durable function locally -before you deploy it. - -## Quick Start - -### A durable function under test +## Quick start ```python -from aws_durable_execution_sdk_python.context import ( +from aws_durable_execution_sdk_python import ( DurableContext, + StepContext, + durable_execution, durable_step, - durable_with_child_context, ) -from aws_durable_execution_sdk_python.execution import durable_execution from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.execution import InvocationStatus +from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner @durable_step -def one(a: int, b: int) -> str: - return f"{a} {b}" - -@durable_step -def two_1(a: int, b: int) -> str: - return f"{a} {b}" +def greet(step_ctx: StepContext, name: str) -> str: + return f"Hello, {name}" -@durable_step -def two_2(a: int, b: int) -> str: - return f"{b} {a}" -@durable_with_child_context -def two(ctx: DurableContext, a: int, b: int) -> str: - two_1_result: str = ctx.step(two_1(a, b)) - two_2_result: str = ctx.step(two_2(a, b)) - return f"{two_1_result} {two_2_result}" +@durable_execution +def handler(event: dict, context: DurableContext) -> str: + greeting = context.step(greet(event["name"])) + context.wait(Duration.from_hours(1), name="cool-down") + return greeting -@durable_step -def three(a: int, b: int) -> str: - return f"{a} {b}" -@durable_execution -def function_under_test(event: Any, context: DurableContext) -> list[str]: - results: list[str] = [] +def test_handler(): + with DurableFunctionTestRunner(handler=handler) as runner: + result = runner.run(input='{"name": "Jane"}', execution_timeout=10) - result_one: str = context.step(one(1, 2)) - results.append(result_one) + assert result.status is InvocationStatus.SUCCEEDED + assert result.result == '"Hello, Jane"' + assert result.get_step("greet").result == '"Hello, Jane"' +``` - context.wait(Duration.from_seconds(1)) +The runner skips waits by default, so the 1-hour wait ends at once. Pass `skip_time=False` to `DurableFunctionTestRunner` to wait in real time. - result_two: str = context.run_in_child_context(two(3, 4)) - results.append(result_two) +`result.result` and each operation's `result` hold the serialized value. `get_step`, `get_wait`, `get_context`, `get_callback`, and `get_invoke` find an operation by name. - result_three: str = context.step(three(5, 6)) - results.append(result_three) +## Common uses - return results -``` +### Complete a callback -### Your test code +Start the execution with `run_async`, then send the callback result: ```python +from aws_durable_execution_sdk_python import DurableContext, durable_execution from aws_durable_execution_sdk_python.execution import InvocationStatus -from aws_durable_execution_sdk_python_testing.runner import ( - ContextOperation, - DurableFunctionTestResult, - DurableFunctionTestRunner, - StepOperation, -) +from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner -def test_my_durable_functions(): - with DurableFunctionTestRunner(handler=function_under_test) as runner: - result: DurableFunctionTestResult = runner.run(input="input str", timeout=10) - assert result.status is InvocationStatus.SUCCEEDED - assert result.result == '["1 2", "3 4 4 3", "5 6"]' +def notify(callback_id: str, context) -> None: + ... # Send callback_id to the approver. - one_result: StepOperation = result.get_step("one") - assert one_result.result == '"1 2"' - two_result: ContextOperation = result.get_context("two") - assert two_result.result == '"3 4 4 3"' +@durable_execution +def approval(event: dict, context: DurableContext) -> dict: + decision = context.wait_for_callback(notify, name="approval") + return {"decision": decision} - three_result: StepOperation = result.get_step("three") - assert three_result.result == '"5 6"' -``` -## Testing functions that invoke other functions -A durable function can call another function with `context.invoke`. The -runner needs to know each target's kind: a durable target runs as a -child durable execution with its own history, while a plain (non-durable) -target is one invocation whose return value is the result. An unknown -target fails the operation with `ResourceNotFoundException`. +def test_approval(): + with DurableFunctionTestRunner(handler=approval) as runner: + execution_arn = runner.run_async(input="{}", execution_timeout=10) + callback_id = runner.wait_for_callback(execution_arn) + runner.send_callback_success(callback_id, b"approved") + result = runner.wait_for_result(execution_arn, timeout=10) -### In-process runner + assert result.status is InvocationStatus.SUCCEEDED + assert result.result == '{"decision": "approved"}' +``` -Register each target with the runner. `register_durable_function` marks a -durable target, and takes its execution timeout and retention; -`register_function` marks a plain one. The runner has one handler per -registered name, not versions: an invoke of `child:prod` runs the handler -registered as `child:prod` if there is one, otherwise the one registered -as `child`. +### Test functions that invoke other functions + +Register each `context.invoke` target with the runner. `register_durable_function` registers a durable target, which runs as a child durable execution. `register_function` registers a plain function, which runs once. An unregistered target fails the invoke with `ResourceNotFoundException`. ```python -from aws_durable_execution_sdk_python.context import DurableContext -from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python import DurableContext, durable_execution from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner + @durable_execution def process_payment(event: dict, context: DurableContext) -> dict: return {"charged": event["amount"]} + def lookup_price(event: dict, context) -> dict: return {"price": 25} + @durable_execution def place_order(event: dict, context: DurableContext) -> dict: price = context.invoke("lookup-price", {"sku": event["sku"]}, name="price") @@ -149,7 +110,8 @@ def place_order(event: dict, context: DurableContext) -> dict: ) return {"sku": event["sku"], "charged": payment["charged"]} -def test_place_order_invokes_both_functions(): + +def test_place_order(): with DurableFunctionTestRunner(handler=place_order) as runner: runner.register_durable_function( "process-payment", process_payment, execution_timeout=60 @@ -161,46 +123,20 @@ def test_place_order_invokes_both_functions(): assert result.get_invoke("payment").status.value == "SUCCEEDED" ``` -## Architecture - -See [docs/architecture.md](docs/architecture.md) for framework -internals. It covers the components, the worker model, the checkpoint -flow, pagination, and a map of the code. - - -## Documentation - -### Error Handling - -A callback failure with omitted diagnostics or an empty `ErrorObject` retains an -empty error object in SDK-facing operation state, in both memory and filesystem -stores. Detailed callback-failure history preserves `Error.Payload: {}` with -`Truncated: false`; history requested without execution data omits the payload and -sets `Truncated: true`. Reconstructing operations from detailed history preserves -that empty object. Missing or null errors remain `None` when decoded. Present -diagnostic fields remain intact, including explicit empty strings and empty -stack-trace lists. The callback still fails and raises the existing -`CallbackExternalError`, including its default message when diagnostics are absent. -Plugin operation information omits wholly empty diagnostics without changing the -stored error. Failed step, child-context, and invoke replay use their existing -default exception message when the stored error has no fields. Timeout history -retains the same data-inclusion rules. - -The testing framework implements AWS-compliant error responses that match the exact format expected by boto3 and AWS services. For detailed information about error response formats, exception types, and troubleshooting, see: +The runner matches a target by name, not by version. An invoke of `child:prod` runs the handler registered as `child:prod` if there is one, and otherwise the one registered as `child`. -- [Error Response Documentation](docs/error-responses.md) +## Advanced -Key features: -- **AWS-compliant JSON format**: Matches boto3 expectations exactly -- **Smithy model compliance**: Field names follow AWS Smithy definitions -- **HTTP status code mapping**: Standard AWS service status codes -- **Boto3 compatibility**: Seamless integration with boto3 error handling +- `DurableFunctionCloudTestRunner(function_name=..., region=...)` runs a test against a deployed function. It returns the same `DurableFunctionTestResult`. +- The local runner returns errors in the same JSON format as the AWS service, so boto3 error handling works the same way. See [docs/error-responses.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md). -## Developers -Please see [CONTRIBUTING.md](CONTRIBUTING.md). It contains the testing guide, sample commands and instructions -for how to contribute to this package. +## Reference -tldr; use `hatch` and it will manage virtual envs and dependencies for you, so you don't have to do it manually. +- [Testing guide](https://docs.aws.amazon.com/durable-execution/testing/) in the AWS Durable Execution documentation. +- [docs/architecture.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-testing/docs/architecture.md): the components, the worker model, the checkpoint flow, and a map of the code. +- [docs/error-responses.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md): error formats, and how callback failures without diagnostics are stored. +- [Source](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-testing/src/aws_durable_execution_sdk_python_testing). +- [Contributing guide](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/CONTRIBUTING.md): the test commands. The repo uses `hatch` to manage virtual environments. ## License diff --git a/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md b/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md index 44a35672e..d21368acd 100644 --- a/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md +++ b/packages/aws-durable-execution-sdk-python-testing/docs/error-responses.md @@ -308,4 +308,20 @@ def test_api_error_response(): 1. **Check exception type**: Verify you're using the correct AWS exception 2. **Validate JSON structure**: Use `to_dict()` to see exact output 3. **Test with boto3**: Verify compatibility with actual boto3 client -4. **Compare with AWS**: Match format with real AWS service responses \ No newline at end of file +4. **Compare with AWS**: Match format with real AWS service responses + +## Callback failures without diagnostics + +A callback failure with omitted diagnostics or an empty `ErrorObject` retains an +empty error object in SDK-facing operation state, in both memory and filesystem +stores. Detailed callback-failure history preserves `Error.Payload: {}` with +`Truncated: false`; history requested without execution data omits the payload and +sets `Truncated: true`. Reconstructing operations from detailed history preserves +that empty object. Missing or null errors remain `None` when decoded. Present +diagnostic fields remain intact, including explicit empty strings and empty +stack-trace lists. The callback still fails and raises the existing +`CallbackExternalError`, including its default message when diagnostics are absent. +Plugin operation information omits wholly empty diagnostics without changing the +stored error. Failed step, child-context, and invoke replay use their existing +default exception message when the stored error has no fields. Timeout history +retains the same data-inclusion rules. diff --git a/packages/aws-durable-execution-sdk-python/README.md b/packages/aws-durable-execution-sdk-python/README.md index dc84924b9..4c4f600d8 100644 --- a/packages/aws-durable-execution-sdk-python/README.md +++ b/packages/aws-durable-execution-sdk-python/README.md @@ -6,291 +6,205 @@ [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/aws/aws-durable-execution-sdk-python/badge)](https://scorecard.dev/viewer/?uri=github.com/aws/aws-durable-execution-sdk-python) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/LICENSE) ------ +Write AWS Lambda durable functions in Python: workflows with checkpointed steps, waits, callbacks, and parallel work. -Build reliable, long-running AWS Lambda workflows with checkpointed steps, waits, callbacks, and parallel execution. +## Install -## ✨ Key Features +```console +pip install aws-durable-execution-sdk-python +``` -- **Automatic checkpointing** - Resume execution after Lambda pauses or restarts -- **Durable steps** - Run work with retry strategies and deterministic replay -- **Waits and callbacks** - Pause for time or external signals without blocking Lambda -- **Parallel and map operations** - Fan out work with configurable completion criteria -- **Child contexts** - Structure complex workflows into isolated subflows -- **Replay-safe logging** - Use `context.logger` for structured, de-duplicated logs -- **Local and cloud testing** - Validate workflows with the testing SDK +## Quick start -## 📦 Packages +```python +from aws_durable_execution_sdk_python import ( + DurableContext, + StepContext, + durable_execution, + durable_step, +) +from aws_durable_execution_sdk_python.config import Duration -| Package | Description | Version | -| --- | --- | --- | -| `aws-durable-execution-sdk-python` | Execution SDK for Lambda durable functions | [![PyPI - Version](https://img.shields.io/pypi/v/aws-durable-execution-sdk-python.svg)](https://pypi.org/project/aws-durable-execution-sdk-python) | -| `aws-durable-execution-sdk-python-testing` | Local/cloud test runner and pytest helpers | [![PyPI - Version](https://img.shields.io/pypi/v/aws-durable-execution-sdk-python-testing.svg)](https://pypi.org/project/aws-durable-execution-sdk-python-testing) | -## Dynamic instrumentation plugins +@durable_step +def validate_order(step_ctx: StepContext, order_id: str) -> dict: + step_ctx.logger.info("Validating order", extra={"order_id": order_id}) + return {"order_id": order_id, "valid": True} -Upgrading an existing application? Follow the [2.x to 3.0 migration guide](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/docs/migration-2.x-to-3.0.md) -for dependency versions, before/after examples, provider packaging, and validation. -Instrumentation plugins can be selected at Lambda cold start without importing -them in the function artifact. Install a provider package in the function or a -Lambda layer, then set an ordered allow-list: +@durable_execution +def handler(event: dict, context: DurableContext) -> dict: + order_id = event["order_id"] + validation = context.step(validate_order(order_id)) + if not validation["valid"]: + return {"status": "rejected", "order_id": order_id} -```text -DURABLE_EXECUTION_PLUGINS=otel-invocation,example_audit + # The function stops while it waits, so no Lambda compute is billed. + context.wait(Duration.from_seconds(5), name="cool-down") + return {"status": "approved", "order_id": order_id} ``` -The SDK resolves those names from the `aws_durable_execution.plugins` Python -entry-point group when the decorated handler is initialized. An unset or blank -variable preserves the existing behavior. The decorator's `plugins` argument -remains supported; explicit factories run first, and a factory passed to the -decorator is not registered a second time through the environment. - -A plugin is registered as a *factory*, not as an instance. A factory is an object -with a `create_plugin(info)` method taking the invocation's `InvocationStartInfo` -and returning a `DurableInstrumentationPlugin`; the SDK calls that method once per -invocation, so the instance it returns serves that one invocation only and can -hold **per-invocation** state in ordinary attributes. - -Per-invocation is narrower than per-execution, and the difference matters. A -durable execution spans as many invocations as it waits, retries or resumes, and -the instance is dropped when each of those returns — so anything a plugin keeps in -its attributes is gone by the next invocation of the same execution. State that -has to survive that has two honest homes: rebuild it from the operation map the -invocation hooks carry (`InvocationStartInfo.operations` is a full snapshot, -which is how the bundled Telemetry exporter reports operations that completed in an -earlier invocation), or put it on the factory, which outlives every invocation — -keyed by execution ARN, and pruned by the owner, because the SDK will not tell the -factory when an execution ends for good. - -Invocation operation-map snapshots and terminal `OperationEndInfo` own a copy of -the error's mutable stack-trace list, including virtual child completions without -a checkpoint. A plugin may modify that list without changing the checkpointed -error, the child error raised to the caller, or a later snapshot. Plugins receiving -the same hook object still share that snapshot; it is not copied for each plugin. - -A factory is an object with a method rather than a plain callable so the -registration type can grow a second, optional member later -- a process-level -flush on execution-environment shutdown, for example -- without a second breaking -change to this surface. - -Write a small factory class and construct the plugin in its `create_plugin`. The -factory holds what outlives an invocation, such as an exporter or a resolved -configuration, and setup work that can fail belongs in the factory's own -constructor rather than the plugin's: +`@durable_execution` makes `handler` the Lambda handler. Each `context.step` result is checkpointed. After a wait or a failure, the function runs again from the start, and completed steps return their checkpointed result. + +## Common uses + +### Wait for an external system ```python -class AuditPluginFactory: - def __init__(self, sink): - self._sink = sink +from aws_durable_execution_sdk_python import DurableContext, durable_execution + - def create_plugin(self, info): - return AuditPlugin(self._sink) +def send_approval_request(callback_id: str, context) -> None: + ... # Send callback_id to the approver. -plugins=[AuditPluginFactory(sink)] +@durable_execution +def handler(event: dict, context: DurableContext) -> dict: + decision = context.wait_for_callback(send_approval_request, name="approval") + return {"decision": decision} ``` -A plugin class is not a factory, and neither is a bare callable. `plugins=[MyPlugin]` -and `plugins=[lambda info: MyPlugin(sink)]` raise `PluginLoadError` during handler -initialization, because neither carries `create_plugin`. A class that declares -`create_plugin` as a `@classmethod` is accepted, since the requirement is the -member and not the kind of object. +The approver completes the callback with the Lambda `SendDurableExecutionCallbackSuccess` API and the callback ID. -Provider packages expose such a factory: +### Process a list of items ```python -from aws_durable_execution_sdk_python.plugin import ( - DurableInstrumentationPlugin, - InvocationStartInfo, -) - +from collections.abc import Sequence -class AuditPlugin(DurableInstrumentationPlugin): - pass +from aws_durable_execution_sdk_python import DurableContext, durable_execution +from aws_durable_execution_sdk_python.config import MapConfig -class AuditPluginFactory: - def create_plugin(self, info: InvocationStartInfo) -> AuditPlugin: - return AuditPlugin() +def resize(ctx: DurableContext, image: str, index: int, images: Sequence[str]) -> str: + return ctx.step(lambda _: f"small-{image}", name=f"resize-{index}") -AUDIT_PLUGIN_FACTORY = AuditPluginFactory() +@durable_execution +def handler(event: dict, context: DurableContext) -> list[str]: + batch = context.map( + event["images"], resize, name="resize", config=MapConfig(max_concurrency=5) + ) + return batch.get_results() ``` -Register the factory instance in the package's `pyproject.toml`: +`context.parallel` runs a list of different functions the same way. -```toml -[project.entry-points."aws_durable_execution.plugins"] -example_audit = "example_audit:AUDIT_PLUGIN_FACTORY" +### Group operations in a child context + +```python +from aws_durable_execution_sdk_python import ( + DurableContext, + durable_execution, + durable_with_child_context, +) +from aws_durable_execution_sdk_python.config import Duration + + +@durable_with_child_context +def ship(ctx: DurableContext, order_id: str) -> str: + label = ctx.step(lambda _: f"label-{order_id}", name="label") + ctx.wait(Duration.from_seconds(1), name="pickup") + return label + + +@durable_execution +def handler(event: dict, context: DurableContext) -> str: + return context.run_in_child_context(ship(event["order_id"]), name="ship") ``` -Provider names must be unique across installed distributions. Missing, -ambiguous, or wrongly shaped providers raise `PluginLoadError` during handler -initialization with the provider and distribution details. - -### Selecting an OpenTelemetry view - -Enable at most one of `InvocationOtelPluginFactory` and -`ExecutionOtelPluginFactory`. This -constraint applies to the combined `plugins=[...]` argument and -`DURABLE_EXECUTION_PLUGINS=otel-invocation` or `otel-execution` selection. -Core 3 with OTel 2 rejects both at cold start with `PluginLoadError` naming the -conflicting views; keep only one. Choose Invocation for work within each Lambda -invocation or Execution for logical operations across the durable execution. -Unrelated instrumentation plugin factories can run alongside either view. -OTel 2 requires core 3.x and the per-invocation factory contract. Release the -coordinated core version before releasing OTel 2. - -Plugin authors explicitly opt in by declaring -`__durable_registration_api__ = 1` on a factory class. That class and its subclasses -have their optional `exclusive_group` metadata validated and their -optional `on_registration_result(registered)` callback invoked. A missing group -or `None` adds no exclusivity constraint. The callback can release rejected -constructor resources and must preserve resources from earlier accepted use. - -The first class declaring the marker in the method resolution order gates -registration and selects the callback contract. An explicit marker other than -the integer `1` shadows an ancestor's opt-in and disables this capability. -Once enabled, every class in the MRO explicitly declaring the integer `1` -contributes its resolved `exclusive_group`. These constraints accumulate: -repeating the marker with a new group adds to inherited groups rather than -replacing them, and `None` does not erase an inherited constraint. A group is -counted once per factory registration even if several ancestors declare it. -Unmarked classes' coincidental same-named legacy fields/helpers remain inert. -Subclasses of the bundled OTel factories therefore retain the view group when they -add their own group. Repeat the marker to add a group or customize the callback; -the callback is still notified once per factory. Factories with no marker anywhere -in their hierarchy retain their existing attributes/helpers. The generic plugin -base defines neither attribute nor hook. The registration API remains independent -of the removed provider API version 1 contract. - -### Invocation worker lifecycle - -The existing `on_invocation_start` and `on_invocation_end` hooks run on the -handler's invocation worker, both in registration order. Start hooks finish -before background checkpoint processing begins. The same worker runs the handler -(including its `finally` blocks), prepares the existing serialized output or -error, checkpoints large results when necessary, joins registered branches while -checkpointing is available, stops and waits for checkpoint processing, then calls -End before returning the outcome. The caller shuts down the handler executor. -There is no separate handler-context hook or context-manager plugin API. - -When plugins are registered, the worker begins with a copy of the caller's -context-variable bindings. Start hooks, the handler, serialization, resource -cleanup and End hooks use that same Context. Changes made by a Start hook remain -visible to later hooks and handler code even if that hook raises; the SDK contains -the exception and retains the plugin without rolling back its bindings. A later -Start hook may replace an earlier binding. Plugins are responsible for restoring -partial bindings themselves when needed, either before raising or during End. -End runs in the same Context as Start, so context-variable tokens retain their -owner. End hooks retain forward registration order, not reverse stack order, -so they must not rely on observing a stack-like unwind of other plugins' contexts. -Each SDK-managed `map` or `parallel` branch admission, including an in-process -resume, receives a fresh copy of the coordinator's bindings when plugins are -registered. Branch changes remain local even when pool threads are reused. -User-created threads retain normal Python context-variable behavior; without -registered plugins, SDK branch submission keeps its existing behavior. -The worker's invocation context is discarded on return, including after plugin -cleanup failures, leaving the host's bindings unchanged. This isolates bindings, -not mutations to shared objects or external side effects. Without plugins, the -handler retains its existing fresh-worker context behavior. - -Core 3 creates a fresh plugin instance per invocation through each registered -factory. OTel 2 requires core 3.x and uses the existing Start/End hooks on those -instances. Upgrade both packages together; core 3 rejects OTel 1.x registrations -and the removed provider API version 1 contract. The independent registration -API remains unchanged. - -### Chained-invoke propagation - -A new invoke START collects synchronous plugin metadata after resolving its -operation and target identity, before checkpointing. The SDK writes a non-blank -`x_amzn_trace_id` contribution to the flat -`ChainedInvokeOptions.XAmznTraceId` member. Function, tenant, payload, operation -name and identity remain unchanged. The member is omitted without a contribution. -Each invoke in a batch has its own metadata; this is not a request-wide header. - -The SDK-owned frozen `PropagationInput` carries `execution_arn`, `operation_id`, -optional `parent_operation_id`, and `target_function_name`. Frozen -`PropagationMetadata` has optional `x_amzn_trace_id`. Neither type depends on -OpenTelemetry or generated service models. The optional synchronous plugin -`provide_propagation_metadata(info)` hook defaults to no contribution. - -The collector keeps the first non-blank opaque value in configured order without -trimming it. Equal values do not conflict; different later values log both plugin -identities and a conflict count. Ordinary hook/getter/result/diagnostic failures -are isolated; cancellation and other `BaseException` control signals keep their -existing behavior. Replaying a checkpointed START, pending operation or terminal -result does not call the hook. If a START was never committed, a later attempt -can collect again; the callback is not exactly-once. - -This requires boto3 1.43.114 or newer, whose botocore dependency includes the -published optional `ChainedInvokeOptions.XAmznTraceId` string field. The real -botocore tests exercise its parameter validation and wire serialization without -field removal, validation bypass or a model-capability fallback. Deployed backend -delivery and downstream topology still require separate validation. -Python has no `DistributedMapOptions` wrapper -or distributed-map START API: its existing map/parallel operations use CONTEXT. -The design's corresponding distributed-map field awaits that future modeled path. - -This release pairs core 3 with OTel 2 and its per-invocation factories. Custom -plugins that omit the optional hook retain their existing tracing behavior. -Backend rollout and deployed downstream trace-topology validation remain pending. - -## 🚀 Quick Start - -Install the execution SDK: +### Call another function -```console -pip install aws-durable-execution-sdk-python +```python +from aws_durable_execution_sdk_python import DurableContext, durable_execution + + +@durable_execution +def handler(event: dict, context: DurableContext) -> dict: + payment = context.invoke("process-payment", {"amount": 25}, name="payment") + return {"charged": payment["charged"]} ``` -Create a durable Lambda handler: +The function stops while it waits for the result. The target can be a durable function or a plain Lambda function. + +## Advanced + +### Retries and errors + +A step retries on failure according to its retry strategy. After the last attempt, the step raises `StepError`. ```python from aws_durable_execution_sdk_python import ( DurableContext, StepContext, + StepError, durable_execution, durable_step, ) -from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.config import Duration, StepConfig +from aws_durable_execution_sdk_python.retries import ( + RetryStrategyConfig, + create_retry_strategy, +) + +RETRY = StepConfig( + retry_strategy=create_retry_strategy( + RetryStrategyConfig(max_attempts=3, initial_delay=Duration.from_seconds(2)) + ) +) + @durable_step -def validate_order(step_ctx: StepContext, order_id: str) -> dict: - step_ctx.logger.info("Validating order", extra={"order_id": order_id}) - return {"order_id": order_id, "valid": True} +def charge(step_ctx: StepContext, amount: int) -> dict: + # Call the payment service here. An exception makes the step retry. + return {"status": "charged", "amount": amount} + @durable_execution def handler(event: dict, context: DurableContext) -> dict: - order_id = event["order_id"] - context.logger.info("Starting workflow", extra={"order_id": order_id}) + try: + return context.step(charge(event["amount"]), config=RETRY) + except StepError as error: + context.logger.warning("Charge failed", extra={"error": str(error)}) + return {"status": "failed"} +``` - validation = context.step(validate_order(order_id), name="validate_order") - if not validation["valid"]: - return {"status": "rejected", "order_id": order_id} +`RetryPresets` in `aws_durable_execution_sdk_python.retries` holds ready-made strategies. - # simulate approval (real world: use wait_for_callback) - context.wait(duration=Duration.from_seconds(5), name="await_confirmation") +### Replay rules - return {"status": "approved", "order_id": order_id} +The function runs again from the start after each wait, retry, or resume. So: + +1. Code outside steps must be deterministic. Put timestamps, random values, and API calls in a step. +2. Do not call durable operations inside a step. Use a child context to group operations. +3. Return values from steps. A step's changes to variables outside it are lost on replay. +4. Use `context.logger` for logs. It does not repeat log lines on replay. + +### Instrumentation plugins + +Set `DURABLE_EXECUTION_PLUGINS` to load installed plugins at cold start, in order: + +```text +DURABLE_EXECUTION_PLUGINS=otel-invocation ``` -## 📚 Documentation +You can also pass plugin factories to `@durable_execution(plugins=[...])`. See [docs/plugins.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python/docs/plugins.md) for the factory contract, the OpenTelemetry views, and trace propagation. -The complete documentation for the AWS Durable Execution SDK for Python lives on the AWS Documentation site: +## Reference -- **[AWS Durable Execution Documentation](https://docs.aws.amazon.com/durable-execution/)** - Concepts, getting started, core operations, advanced topics, and API reference -- **[AWS Lambda Durable Functions Guide](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html)** - How durable functions work on Lambda +- [AWS Durable Execution documentation](https://docs.aws.amazon.com/durable-execution/): concepts, operations, and API reference. +- [Lambda durable functions guide](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html): deployment, permissions, and quotas. +- [docs/plugins.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python/docs/plugins.md): instrumentation plugin contract. +- [Migration guide, 2.x to 3.0](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/docs/migration-2.x-to-3.0.md). +- [Testing SDK](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-testing): run durable functions in local tests. +- [Source](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python). -## 💬 Feedback & Support +## Feedback and support - [Bug report](https://github.com/aws/aws-durable-execution-sdk-python/issues/new?template=bug_report.yml) - [Feature request](https://github.com/aws/aws-durable-execution-sdk-python/issues/new?template=feature_request.yml) - [Documentation feedback](https://github.com/aws/aws-durable-execution-sdk-python/issues/new?template=documentation.yml) -- [Contributing guide](CONTRIBUTING.md) +- [Contributing guide](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/CONTRIBUTING.md) -## 📄 License +## License See the [LICENSE](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/LICENSE) file for our project's licensing. diff --git a/packages/aws-durable-execution-sdk-python/docs/plugins.md b/packages/aws-durable-execution-sdk-python/docs/plugins.md new file mode 100644 index 000000000..474ac1ff3 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python/docs/plugins.md @@ -0,0 +1,213 @@ +# Instrumentation plugins + +This file describes how the SDK loads instrumentation plugins, and the +contract that a plugin and its factory follow. + +Upgrading an existing application? Follow the [2.x to 3.0 migration guide](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/docs/migration-2.x-to-3.0.md) +for dependency versions, before/after examples, provider packaging, and validation. + +Instrumentation plugins can be selected at Lambda cold start without importing +them in the function artifact. Install a provider package in the function or a +Lambda layer, then set an ordered allow-list: + +```text +DURABLE_EXECUTION_PLUGINS=otel-invocation,example_audit +``` + +The SDK resolves those names from the `aws_durable_execution.plugins` Python +entry-point group when the decorated handler is initialized. An unset or blank +variable preserves the existing behavior. The decorator's `plugins` argument +remains supported; explicit factories run first, and a factory passed to the +decorator is not registered a second time through the environment. + +A plugin is registered as a *factory*, not as an instance. A factory is an object +with a `create_plugin(info)` method taking the invocation's `InvocationStartInfo` +and returning a `DurableInstrumentationPlugin`; the SDK calls that method once per +invocation, so the instance it returns serves that one invocation only and can +hold **per-invocation** state in ordinary attributes. + +Per-invocation is narrower than per-execution, and the difference matters. A +durable execution spans as many invocations as it waits, retries or resumes, and +the instance is dropped when each of those returns — so anything a plugin keeps in +its attributes is gone by the next invocation of the same execution. State that +has to survive that has two honest homes: rebuild it from the operation map the +invocation hooks carry (`InvocationStartInfo.operations` is a full snapshot, +which is how the bundled Telemetry exporter reports operations that completed in an +earlier invocation), or put it on the factory, which outlives every invocation — +keyed by execution ARN, and pruned by the owner, because the SDK will not tell the +factory when an execution ends for good. + +Invocation operation-map snapshots and terminal `OperationEndInfo` own a copy of +the error's mutable stack-trace list, including virtual child completions without +a checkpoint. A plugin may modify that list without changing the checkpointed +error, the child error raised to the caller, or a later snapshot. Plugins receiving +the same hook object still share that snapshot; it is not copied for each plugin. + +A factory is an object with a method rather than a plain callable so the +registration type can grow a second, optional member later -- a process-level +flush on execution-environment shutdown, for example -- without a second breaking +change to this surface. + +Write a small factory class and construct the plugin in its `create_plugin`. The +factory holds what outlives an invocation, such as an exporter or a resolved +configuration, and setup work that can fail belongs in the factory's own +constructor rather than the plugin's: + +```python +class AuditPluginFactory: + def __init__(self, sink): + self._sink = sink + + def create_plugin(self, info): + return AuditPlugin(self._sink) + + +plugins=[AuditPluginFactory(sink)] +``` + +A plugin class is not a factory, and neither is a bare callable. `plugins=[MyPlugin]` +and `plugins=[lambda info: MyPlugin(sink)]` raise `PluginLoadError` during handler +initialization, because neither carries `create_plugin`. A class that declares +`create_plugin` as a `@classmethod` is accepted, since the requirement is the +member and not the kind of object. + +Provider packages expose such a factory: + +```python +from aws_durable_execution_sdk_python.plugin import ( + DurableInstrumentationPlugin, + InvocationStartInfo, +) + + +class AuditPlugin(DurableInstrumentationPlugin): + pass + + +class AuditPluginFactory: + def create_plugin(self, info: InvocationStartInfo) -> AuditPlugin: + return AuditPlugin() + + +AUDIT_PLUGIN_FACTORY = AuditPluginFactory() +``` + +Register the factory instance in the package's `pyproject.toml`: + +```toml +[project.entry-points."aws_durable_execution.plugins"] +example_audit = "example_audit:AUDIT_PLUGIN_FACTORY" +``` + +Provider names must be unique across installed distributions. Missing, +ambiguous, or wrongly shaped providers raise `PluginLoadError` during handler +initialization with the provider and distribution details. + +## Selecting an OpenTelemetry view + +Enable at most one of `InvocationOtelPluginFactory` and +`ExecutionOtelPluginFactory`. This +constraint applies to the combined `plugins=[...]` argument and +`DURABLE_EXECUTION_PLUGINS=otel-invocation` or `otel-execution` selection. +Core 3 with OTel 2 rejects both at cold start with `PluginLoadError` naming the +conflicting views; keep only one. Choose Invocation for work within each Lambda +invocation or Execution for logical operations across the durable execution. +Unrelated instrumentation plugin factories can run alongside either view. +OTel 2 requires core 3.x and the per-invocation factory contract. Release the +coordinated core version before releasing OTel 2. + +Plugin authors explicitly opt in by declaring +`__durable_registration_api__ = 1` on a factory class. That class and its subclasses +have their optional `exclusive_group` metadata validated and their +optional `on_registration_result(registered)` callback invoked. A missing group +or `None` adds no exclusivity constraint. The callback can release rejected +constructor resources and must preserve resources from earlier accepted use. + +The first class declaring the marker in the method resolution order gates +registration and selects the callback contract. An explicit marker other than +the integer `1` shadows an ancestor's opt-in and disables this capability. +Once enabled, every class in the MRO explicitly declaring the integer `1` +contributes its resolved `exclusive_group`. These constraints accumulate: +repeating the marker with a new group adds to inherited groups rather than +replacing them, and `None` does not erase an inherited constraint. A group is +counted once per factory registration even if several ancestors declare it. +Unmarked classes' coincidental same-named legacy fields/helpers remain inert. +Subclasses of the bundled OTel factories therefore retain the view group when they +add their own group. Repeat the marker to add a group or customize the callback; +the callback is still notified once per factory. Factories with no marker anywhere +in their hierarchy retain their existing attributes/helpers. The generic plugin +base defines neither attribute nor hook. The registration API remains independent +of the removed provider API version 1 contract. + +## Invocation worker lifecycle + +The existing `on_invocation_start` and `on_invocation_end` hooks run on the +handler's invocation worker, both in registration order. Start hooks finish +before background checkpoint processing begins. The same worker runs the handler +(including its `finally` blocks), prepares the existing serialized output or +error, checkpoints large results when necessary, joins registered branches while +checkpointing is available, stops and waits for checkpoint processing, then calls +End before returning the outcome. The caller shuts down the handler executor. +There is no separate handler-context hook or context-manager plugin API. + +When plugins are registered, the worker begins with a copy of the caller's +context-variable bindings. Start hooks, the handler, serialization, resource +cleanup and End hooks use that same Context. Changes made by a Start hook remain +visible to later hooks and handler code even if that hook raises; the SDK contains +the exception and retains the plugin without rolling back its bindings. A later +Start hook may replace an earlier binding. Plugins are responsible for restoring +partial bindings themselves when needed, either before raising or during End. +End runs in the same Context as Start, so context-variable tokens retain their +owner. End hooks retain forward registration order, not reverse stack order, +so they must not rely on observing a stack-like unwind of other plugins' contexts. +Each SDK-managed `map` or `parallel` branch admission, including an in-process +resume, receives a fresh copy of the coordinator's bindings when plugins are +registered. Branch changes remain local even when pool threads are reused. +User-created threads retain normal Python context-variable behavior; without +registered plugins, SDK branch submission keeps its existing behavior. +The worker's invocation context is discarded on return, including after plugin +cleanup failures, leaving the host's bindings unchanged. This isolates bindings, +not mutations to shared objects or external side effects. Without plugins, the +handler retains its existing fresh-worker context behavior. + +Core 3 creates a fresh plugin instance per invocation through each registered +factory. OTel 2 requires core 3.x and uses the existing Start/End hooks on those +instances. Upgrade both packages together; core 3 rejects OTel 1.x registrations +and the removed provider API version 1 contract. The independent registration +API remains unchanged. + +## Chained-invoke propagation + +A new invoke START collects synchronous plugin metadata after resolving its +operation and target identity, before checkpointing. The SDK writes a non-blank +`x_amzn_trace_id` contribution to the flat +`ChainedInvokeOptions.XAmznTraceId` member. Function, tenant, payload, operation +name and identity remain unchanged. The member is omitted without a contribution. +Each invoke in a batch has its own metadata; this is not a request-wide header. + +The SDK-owned frozen `PropagationInput` carries `execution_arn`, `operation_id`, +optional `parent_operation_id`, and `target_function_name`. Frozen +`PropagationMetadata` has optional `x_amzn_trace_id`. Neither type depends on +OpenTelemetry or generated service models. The optional synchronous plugin +`provide_propagation_metadata(info)` hook defaults to no contribution. + +The collector keeps the first non-blank opaque value in configured order without +trimming it. Equal values do not conflict; different later values log both plugin +identities and a conflict count. Ordinary hook/getter/result/diagnostic failures +are isolated; cancellation and other `BaseException` control signals keep their +existing behavior. Replaying a checkpointed START, pending operation or terminal +result does not call the hook. If a START was never committed, a later attempt +can collect again; the callback is not exactly-once. + +This requires boto3 1.43.114 or newer, whose botocore dependency includes the +published optional `ChainedInvokeOptions.XAmznTraceId` string field. The real +botocore tests exercise its parameter validation and wire serialization without +field removal, validation bypass or a model-capability fallback. Deployed backend +delivery and downstream topology still require separate validation. +Python has no `DistributedMapOptions` wrapper +or distributed-map START API: its existing map/parallel operations use CONTEXT. +The design's corresponding distributed-map field awaits that future modeled path. + +This release pairs core 3 with OTel 2 and its per-invocation factories. Custom +plugins that omit the optional hook retain their existing tracing behavior. +Backend rollout and deployed downstream trace-topology validation remain pending.