diff --git a/packages/aws-durable-execution-sdk-python-compute-worker/README.md b/packages/aws-durable-execution-sdk-python-compute-worker/README.md index b9e5c1e36..19283b852 100644 --- a/packages/aws-durable-execution-sdk-python-compute-worker/README.md +++ b/packages/aws-durable-execution-sdk-python-compute-worker/README.md @@ -2,7 +2,7 @@ > This package is experimental. Its API can change in any release. -The compute worker runs jobs from durable functions on compute that is defined outside the durable function. Each compute type has its own subpackage. Today the package has one subpackage: `aws_durable_execution_sdk_python_compute_worker.microvm`, the worker for AWS Lambda MicroVMs. The rest of this README describes the MicroVM worker. +The compute worker runs jobs from durable functions on compute that is defined outside the durable function. Each compute type has its own subpackage: `aws_durable_execution_sdk_python_compute_worker.microvm`, the worker for AWS Lambda MicroVMs, and `aws_durable_execution_sdk_python_compute_worker.agentcore`, the worker for Amazon Bedrock AgentCore Runtime. The rest of this README describes the MicroVM worker, except the section [Run a job in AgentCore Runtime](#run-a-job-in-agentcore-runtime). Run jobs from a durable function inside an AWS Lambda MicroVM, and report each job's result to the durable function. @@ -161,6 +161,59 @@ With `auto_suspend_on_idle`, the role also needs `lambda:SuspendMicrovm` on the The worker needs boto3 1.43.35 or later. +## Run a job in AgentCore Runtime + +> The `agentcore` subpackage is experimental. Its API can change in any release. + +`start_agentcore_worker` serves the [AgentCore Runtime HTTP contract](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-http-protocol-contract.html) in the agent's container. It needs no dependency beyond this package. + +```python +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + AgentcoreJobContext, + start_agentcore_worker, +) + + +def review(job_input: dict, context: AgentcoreJobContext) -> dict: + context.logger.info("reviewing", extra={"pr": job_input["pr"]}) + return {"approved": True, "comments": []} + + +if __name__ == "__main__": + start_agentcore_worker(handler=review).wait() +``` + +The durable function sends the job with the `agentcore` operation of the [extras](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-extras#run-a-job-in-agentcore-runtime) package: + +```python +review = agentcore(context, "review", job_input={"pr": 7}, config=AGENT) +``` + +The worker listens on port 8080, because AgentCore Runtime sends its requests to that port. The host depends on where the worker runs: + +- Inside AgentCore Runtime, the `AGENTCORE_RUNTIME_URL` environment variable is set. The worker then listens on `0.0.0.0`, as the AgentCore contract requires. The runtime publishes no port, and AgentCore authorizes each `InvokeAgentRuntime` caller before it forwards the request. +- Anywhere else, such as a laptop, the worker listens on `127.0.0.1`. The worker has no authentication, so it must not be reachable from the network. +- A `host` argument overrides both. + +- `POST /invocations` reads the job document, starts the job's thread, and answers `{"accepted": true}` at once. The thread runs the handler only after the answer is written. A callback ID that the worker already accepted gets the same answer, and the job does not run again. A document that does not match the contract gets HTTP 400. +- `GET /ping` answers `{"status": "HealthyBusy"}` while a job runs, and `{"status": "Healthy"}` otherwise. `time_of_last_update` holds the Unix time, in seconds, of the worker's start or of the last change of the status. It does not change between pings with the same status. +- The job thread sends heartbeats when the job sets a heartbeat timeout, and then the success or failure callback. + +The worker remembers the callback IDs of accepted jobs only in its process memory. So it runs a job once per process. An answer can be lost after the job finished, and the retry can arrive after the idle timeout replaced the session's process. The new process then runs the same job again. So delivery is at-least-once across process replacement. Make the handler's side effects safe to repeat. + +While a job runs, the worker answers `/ping` with `HealthyBusy`, so the agent runtime's idle timeout (`idleRuntimeSessionTimeout`) does not end the job. After the idle timeout, a session with no running job ends. A later request with the same session ID then gets a new microVM without an error, and the memory and files of the old one are lost. The `agentcore` operation sends one job per session, so this loss cannot affect a job. + +### Rules for AgentCore jobs + +1. Set `heartbeat_timeout`. It ends the job when the worker crashes or stops reporting. The first heartbeat period starts when the callback is created, before the job is delivered. So it must cover the request step with its retries and a cold session start, which took 1.4 to 3.1 seconds in a live test. 30 seconds or more is a safe value. +2. `timeout` must fit the agent runtime's configured `maxLifetime`. +3. A job can run more than once when AgentCore replaces the worker process. Make handler side effects safe to repeat. +4. Your logger must not throw. +5. The runtime's idle timeout applies only while no job is running. One `agentcore()` call runs one job, so it does not lose state between jobs. +6. Give the durable function a timeout of at least 11 seconds. Each AgentCore call needs the 10-second invocation reserve and 1 second for the call. + +The agent runtime's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the durable function's ARN. The function's execution role needs `bedrock-agentcore:InvokeAgentRuntime` and `bedrock-agentcore:StopRuntimeSession` on the agent runtime ARN and on `/runtime-endpoint/*`. + ## Reference - [docs/worker.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-compute-worker/docs/worker.md): the request flow, the lifecycle hooks, idle suspend outcomes, the job documents, reporting, heartbeats, logging fields, and best practices. diff --git a/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/__init__.py b/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/__init__.py new file mode 100644 index 000000000..a22bad076 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/__init__.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""Runs jobs from AWS Lambda durable functions inside an Amazon Bedrock +AgentCore Runtime session. + +This module is experimental. Its API can change in any release. + +The exports match ``@aws/durable-execution-sdk-js-compute-worker/agentcore``. +""" + +from aws_durable_execution_sdk_python_compute_worker.agentcore.worker import ( + AGENTCORE_HOST, + AGENTCORE_PORT, + INVOCATIONS_PATH, + LOCAL_HOST, + PING_PATH, + RUNTIME_URL_ENV, + AgentcoreJobContext, + AgentcoreJobHandler, + AgentcoreWorker, + default_host, + start_agentcore_worker, +) + + +__all__ = [ + "AGENTCORE_HOST", + "AGENTCORE_PORT", + "INVOCATIONS_PATH", + "LOCAL_HOST", + "PING_PATH", + "RUNTIME_URL_ENV", + "AgentcoreJobContext", + "AgentcoreJobHandler", + "AgentcoreWorker", + "default_host", + "start_agentcore_worker", +] diff --git a/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/worker.py b/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/worker.py new file mode 100644 index 000000000..41259f6e5 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-compute-worker/src/aws_durable_execution_sdk_python_compute_worker/agentcore/worker.py @@ -0,0 +1,603 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The HTTP server that receives jobs inside an AgentCore Runtime session. + +AgentCore Runtime sends each ``InvokeAgentRuntime`` request to +``POST /invocations`` in the agent's container, and checks the session with +``GET /ping``. This module answers both, runs each job in its own thread, and +reports the job's outcome to its callback. + +The callback reporter, the heartbeats, and the job document parser come from +the MicroVM worker. The job document is the MicroVM job request without +``microvmId``. +""" + +from __future__ import annotations + +import inspect +import logging +import os +import socketserver +import sys +import threading +import time +from collections import OrderedDict +from collections.abc import Callable +from dataclasses import dataclass +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import TYPE_CHECKING, Any + +from aws_durable_execution_sdk_python_compute_worker.microvm.callback_reporter import ( + CallbackReporter, + ResultSerializationError, + ResultTooLargeError, + is_terminal_callback_error, +) +from aws_durable_execution_sdk_python_compute_worker.microvm.heartbeats import ( + Heartbeats, + validate_heartbeat_interval, +) +from aws_durable_execution_sdk_python_compute_worker.microvm.logger import ( + MicrovmWorkerLogger, + describe, + safe_logger, +) +from aws_durable_execution_sdk_python_compute_worker.microvm.payload import ( + InvalidRunHookPayloadError, + MicrovmJobDocument, + MicrovmJobRequest, +) +from aws_durable_execution_sdk_python_compute_worker.microvm.worker import ( + MAX_FINISHED_JOBS, + MAX_JOB_BODY_BYTES, + _Exchange, +) + + +if TYPE_CHECKING: + from mypy_boto3_lambda import LambdaClient +else: + LambdaClient = Any + + +AGENTCORE_PORT = 8080 +"""AgentCore Runtime sends requests to port 8080 of the agent's container, +unless the agent runtime's ``protocolConfiguration`` sets another port.""" + +AGENTCORE_HOST = "0.0.0.0" # noqa: S104 +"""The address that the worker listens on inside AgentCore Runtime. + +The AgentCore Runtime service contract requires the agent to listen on +``0.0.0.0:8080``. The runtime publishes no port. Only the AgentCore data +plane reaches it, after it has authorized the ``InvokeAgentRuntime`` caller. +So the worker listens on all interfaces only inside AgentCore Runtime. +""" + +LOCAL_HOST = "127.0.0.1" +"""The address that the worker listens on outside AgentCore Runtime. + +The worker has no authentication. On a laptop or a dev host, a bind on all +interfaces would let any machine on the network send it a job. So outside +AgentCore Runtime the worker listens on loopback only. +""" + +RUNTIME_URL_ENV = "AGENTCORE_RUNTIME_URL" +"""The environment variable that marks the AgentCore Runtime environment. + +A live run in a runtime with direct code deployment found this variable +set, and found no ``/.dockerenv`` file and no ``DOCKER_CONTAINER`` +variable. So the worker uses this variable, not the Docker markers, to +choose the default host. +""" + + +def default_host() -> str: + """The default host: :data:`AGENTCORE_HOST` when + :data:`RUNTIME_URL_ENV` is set, and :data:`LOCAL_HOST` otherwise.""" + return AGENTCORE_HOST if os.environ.get(RUNTIME_URL_ENV) else LOCAL_HOST + + +INVOCATIONS_PATH = "/invocations" +"""The path of the ``InvokeAgentRuntime`` requests.""" + +PING_PATH = "/ping" +"""The path of AgentCore's health checks.""" + +SESSION_ID_HEADER = "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id" +"""The header in which AgentCore passes the runtime session ID.""" + +DEFAULT_LOGGER_NAME = "aws_durable_execution_sdk_python_compute_worker.agentcore" +"""The name of the standard library logger that the worker uses by default.""" + +_SERVER_POLL_SECONDS = 0.1 + + +def _now() -> int: + """The current Unix time in whole seconds. Tests replace it.""" + return int(time.time()) + + +_REQUEST_TIMEOUT_SECONDS = 60.0 + + +@dataclass(frozen=True, kw_only=True) +class AgentcoreJobContext: + """What the job handler receives besides its input.""" + + callback_id: str + """The callback ID that the worker completes with the handler's result.""" + session_id: str | None + """The AgentCore runtime session ID, from the request header. ``None`` + when the request had no such header.""" + region: str + """The Region of the durable function.""" + cancelled: threading.Event + """Set when a heartbeat finds that the durable function no longer + accepts the result, for example after a timeout. + + The worker learns it only from a heartbeat answer. So it is set only for + a job with a heartbeat timeout. Python cannot stop a running thread. So + the handler should check the event and stop its work when it is set. + """ + logger: MicrovmWorkerLogger + """The worker's logger.""" + + +AgentcoreJobHandler = Callable[[Any, AgentcoreJobContext], Any] +"""The job to run. + +The handler receives the job's input and an :class:`AgentcoreJobContext`. +Its return value becomes the durable function's result, and must be +JSON-serializable. An exception fails the durable operation with the +exception's class name and message. +""" + + +class _JobRunner: + """Answers the requests of AgentCore Runtime, and runs the jobs. + + :func:`start_agentcore_worker` describes the requests and the answers. + """ + + def __init__( + self, + *, + handler: AgentcoreJobHandler, + create_client: Callable[[str], LambdaClient] | None, + heartbeat_interval_seconds: float | None, + logger: MicrovmWorkerLogger, + ) -> None: + self._handler = handler + self._create_client = create_client + self._heartbeat_interval_seconds = heartbeat_interval_seconds + self._logger = logger + self._lock = threading.Lock() + self._changed = threading.Condition(self._lock) + self._running: set[str] = set() + # The accepted callback IDs whose jobs have ended, oldest first. + self._finished: OrderedDict[str, None] = OrderedDict() + # The Unix time, in seconds, of the last change between Healthy and + # HealthyBusy. The worker's start counts as a change. + self._last_status_change = _now() + + def ping(self) -> dict[str, object]: + """The answer to ``GET /ping``. + + AgentCore documents ``time_of_last_update`` as optional. It must + hold the time of the last status change, not the time of the ping. + The issue aws/bedrock-agentcore-sdk-python#471 reports that a + ``HealthyBusy`` session without the field can end at the idle + timeout. So the answer always carries the field, and the field + changes only when the status changes. + """ + with self._lock: + return { + "status": "HealthyBusy" if self._running else "Healthy", + "time_of_last_update": self._last_status_change, + } + + def idle(self, timeout: float | None) -> bool: + """Wait until no job runs or reports its outcome.""" + with self._changed: + return self._changed.wait_for(lambda: not self._running, timeout) + + def handle(self, request: BaseHTTPRequestHandler) -> None: + """Answer one request.""" + exchange = _Exchange(request, self._logger) + try: + if exchange.method == "GET" and exchange.path == PING_PATH: + exchange.respond(200, self.ping()) + elif exchange.method == "POST" and exchange.path == INVOCATIONS_PATH: + self._handle_invocation(exchange) + else: + exchange.respond(404, {"error": "not found"}) + except Exception as error: # noqa: BLE001 + self._logger.error( + "could not handle the request", extra={"error": describe(error)} + ) + exchange.respond(500, {"error": "internal error"}) + + def _handle_invocation(self, exchange: _Exchange) -> None: + try: + body = exchange.read_json(MAX_JOB_BODY_BYTES) + request = MicrovmJobRequest.from_dict(body) + except InvalidRunHookPayloadError as error: + # A 400 makes InvokeAgentRuntime fail. The default retry strategy + # does not retry a 4xx answer. So the durable function fails at + # once with AgentcoreDeliveryError. + self._logger.warning( + "rejected an invalid job request", extra={"error": describe(error)} + ) + exchange.respond(400, {"error": str(error)}) + return + job = request.job + with self._lock: + known = job.callback_id in self._running or job.callback_id in ( + self._finished + ) + if not known: + if not self._running: + self._last_status_change = _now() + self._running.add(job.callback_id) + if known: + self._logger.info( + "ignored a repeated job", extra={"callbackId": job.callback_id} + ) + exchange.respond(200, {"accepted": True}) + return + session_id = exchange.header(SESSION_ID_HEADER) + # The thread starts before the answer, so a failed start can answer + # 500. The thread waits on this event before it runs the handler. So + # a handler that holds the GIL does not delay the answer. + answered = threading.Event() + try: + threading.Thread( + target=self._run_job, + args=(job, request.region, session_id, answered), + name=f"agentcore-job-{job.callback_id[:16]}", + daemon=True, + ).start() + except Exception as error: # noqa: BLE001 + # The job never ran. The callback ID is forgotten, so that a + # later delivery of the same job can run it. + self._logger.error( + "could not start the job's thread", + extra={"callbackId": job.callback_id, "error": describe(error)}, + ) + self._end_job(job.callback_id, finished=False) + exchange.respond(500, {"error": "could not start the job"}) + return + try: + exchange.respond(200, {"accepted": True}) + finally: + answered.set() + + def _end_job(self, callback_id: str, *, finished: bool) -> None: + with self._changed: + self._running.discard(callback_id) + if not self._running: + self._last_status_change = _now() + if finished: + self._finished[callback_id] = None + if len(self._finished) > MAX_FINISHED_JOBS: + self._finished.popitem(last=False) + self._changed.notify_all() + + def _run_job( + self, + job: MicrovmJobDocument, + region: str, + session_id: str | None, + answered: threading.Event, + ) -> None: + answered.wait() + fields: dict[str, object] = { + "callbackId": job.callback_id, + "runtimeSessionId": session_id, + } + try: + self._run_job_and_report(job, region, session_id, fields) + except Exception as error: # noqa: BLE001 + self._logger.error( + "job failed inside the worker", + extra={**fields, "error": describe(error)}, + ) + finally: + self._end_job(job.callback_id, finished=True) + + def _run_job_and_report( + self, + job: MicrovmJobDocument, + region: str, + session_id: str | None, + fields: dict[str, object], + ) -> None: + reporter = ( + CallbackReporter.create(job.callback_id, region, logger=self._logger) + if self._create_client is None + else CallbackReporter( + job.callback_id, self._create_client(region), logger=self._logger + ) + ) + try: + cancelled = threading.Event() + try: + heartbeats = Heartbeats.start( + reporter, + heartbeat_timeout_seconds=job.heartbeat_timeout_seconds, + on_callback_gone=lambda _error: cancelled.set(), + heartbeat_interval_seconds=self._heartbeat_interval_seconds, + logger=self._logger, + ) + except Exception as error: # noqa: BLE001 + # The job has its answer, so no later delivery runs it. Fail + # the callback, as the MicroVM worker does, instead of leaving + # the job to its heartbeat timeout or its timeout. + self._logger.error( + "could not start the job's heartbeats", + extra={**fields, "error": describe(error)}, + ) + self._report(reporter, None, error, fields) + return + try: + self._logger.info("job started", extra=fields) + context = AgentcoreJobContext( + callback_id=job.callback_id, + session_id=session_id, + region=region, + cancelled=cancelled, + logger=self._logger, + ) + value: Any = None + failure: BaseException | None = None + try: + value = self._handler(job.input, context) + except BaseException as error: # noqa: BLE001 + # SystemExit from the handler would otherwise end the + # thread without a report. + failure = error + # Heartbeats keep running while the outcome is reported. A + # completion attempt can stall for up to 30 seconds. + if heartbeats.handler_settled(): + self._report(reporter, value, failure, fields) + else: + self._logger.warning( + "job outcome not reported: the callback is gone", + extra=fields, + ) + finally: + heartbeats.stop() + finally: + reporter.close() + + def _report( + self, + reporter: CallbackReporter, + value: Any, + failure: BaseException | None, + fields: dict[str, object], + ) -> None: + try: + if failure is None: + try: + reporter.succeed(value) + except (ResultTooLargeError, ResultSerializationError) as error: + # The result was never sent. Report it as a failure, so + # the durable function learns why instead of waiting for + # its timeout. + reporter.fail(error) + self._logger.info( + "job failed: its result cannot be sent", + extra={**fields, "error": describe(error)}, + ) + return + self._logger.info("job succeeded", extra=fields) + return + reporter.fail(failure) + self._logger.info( + "job failed", extra={**fields, "error": describe(failure)} + ) + except Exception as error: # noqa: BLE001 + self._logger.error( + "job outcome could not be reported", + extra={ + **fields, + "terminal": is_terminal_callback_error(error), + "error": describe(error), + }, + ) + + +def start_agentcore_worker( + *, + handler: AgentcoreJobHandler, + logger: MicrovmWorkerLogger | None = None, + create_client: Callable[[str], LambdaClient] | None = None, + heartbeat_interval_seconds: float | None = None, + host: str | None = None, + port: int = AGENTCORE_PORT, +) -> AgentcoreWorker: + """Start the HTTP server that AgentCore Runtime sends the jobs to. + + The server runs in a daemon thread. So end the container's entry point + with :meth:`AgentcoreWorker.wait`. + + On ``POST /invocations`` the worker: + + 1. Parses the job document, and checks its ``version``. A document that + does not match the contract gets HTTP 400. + 2. Answers ``{"accepted": true}`` for a callback ID that it has already + accepted, and does nothing else. ``InvokeAgentRuntime`` has no + idempotency token, so a retried request can deliver the same job + again. The worker remembers the last 1,000 finished callback IDs, + in process memory only. So a job runs once per process. A new + process after the idle timeout runs a repeated job again, so the + handler's side effects must be idempotent. + 3. Otherwise starts the job's thread, and answers + ``{"accepted": true}``. The thread runs the handler only after the + answer is written. So a handler that holds the GIL does not delay + the answer. A thread that cannot start gets HTTP 500, and the + callback ID is forgotten, so that a later delivery can run the job. + + The job's thread sends heartbeats when the job has a heartbeat timeout, + and then the success or the failure callback. + + ``GET /ping`` answers ``{"status": "HealthyBusy"}`` while a job runs or + reports, and ``{"status": "Healthy"}`` otherwise. AgentCore keeps a + ``HealthyBusy`` session alive. The answer also carries + ``time_of_last_update``: the Unix time, in seconds, of the worker's start + or of the last change between the two statuses. Every other request gets HTTP 404. + + Args: + handler: The job to run for each job request. + logger: Receives the worker's log lines. Defaults to the standard + library logger ``aws_durable_execution_sdk_python_compute_worker.agentcore``. + create_client: Creates the Lambda client for the callback calls of + one job, from the durable function's Region. Defaults to a boto3 + client with the default credential chain, which resolves the + agent runtime's execution role. + heartbeat_interval_seconds: An explicit heartbeat interval. It is cut + to a third of the job's heartbeat timeout. + host: The address to listen on. ``None`` picks ``0.0.0.0``, which + the AgentCore Runtime service contract requires, when the + ``AGENTCORE_RUNTIME_URL`` environment variable is set. Otherwise + it picks ``127.0.0.1``. See :func:`default_host`. + port: The port to listen on. Defaults to 8080, the port that + AgentCore Runtime sends to. 0 picks a free port. + + Raises: + TypeError: When ``handler`` is not a plain function. + ValueError: When ``heartbeat_interval_seconds`` is not above 0 and at + most 900. + OSError: When the port cannot be bound. + """ + if not callable(handler): + msg = f"a job handler must be callable. Got {type(handler).__name__}." + raise TypeError(msg) + # An async function returns a coroutine, which the worker would report + # as a result that is not JSON-serializable. So it is refused here. + if inspect.iscoroutinefunction(handler) or inspect.iscoroutinefunction( + getattr(type(handler), "__call__", None) + ): + msg = "a job handler must be a plain function, not an async function" + raise TypeError(msg) + validate_heartbeat_interval(heartbeat_interval_seconds) + runner = _JobRunner( + handler=handler, + create_client=create_client, + heartbeat_interval_seconds=heartbeat_interval_seconds, + logger=safe_logger( + logger if logger is not None else logging.getLogger(DEFAULT_LOGGER_NAME) + ), + ) + server = _Server((default_host() if host is None else host, port), runner) + threading.Thread( + target=server.serve_forever, + kwargs={"poll_interval": _SERVER_POLL_SECONDS}, + name="agentcore-worker-server", + daemon=True, + ).start() + return AgentcoreWorker(runner=runner, server=server) + + +class AgentcoreWorker: + """A running worker server. + + Create it with :func:`start_agentcore_worker`. See that function for the + requests that the worker answers. + """ + + def __init__(self, *, runner: _JobRunner, server: ThreadingHTTPServer) -> None: + self._runner = runner + self._server = server + self._lock = threading.Lock() + self._closed = threading.Event() + + @property + def port(self) -> int: + """The port the server listens on.""" + return int(self._server.server_address[1]) + + def idle(self, timeout: float | None = None) -> bool: + """Wait until every job started so far has reported its outcome. + + Returns: + ``True`` when no job runs. ``False`` when the timeout ended first. + """ + return self._runner.idle(timeout) + + def wait(self, timeout: float | None = None) -> bool: + """Block until :meth:`close` is called. + + The server runs in a daemon thread, so the process would otherwise + end when the main thread ends. Call this method at the end of the + container's entry point. + + Returns: + ``True`` when the worker is closed. ``False`` when the timeout + ended first. + """ + return self._closed.wait(timeout) + + def close(self) -> None: + """Stop accepting requests. Running jobs continue.""" + with self._lock: + if self._closed.is_set(): + return + self._server.shutdown() + self._server.server_close() + self._closed.set() + + +class _RequestHandler(BaseHTTPRequestHandler): + """Passes every GET and POST request to the server's :class:`_JobRunner`.""" + + server: _Server + timeout = _REQUEST_TIMEOUT_SECONDS + # HTTP/1.1, as the MicroVM worker. Every answer closes its connection. + protocol_version = "HTTP/1.1" + + def do_GET(self) -> None: # noqa: N802 + self.server.runner.handle(self) + + def do_POST(self) -> None: # noqa: N802 + self.server.runner.handle(self) + + def log_message(self, format: str, *args: Any) -> None: # noqa: A002 + # The base class writes every request to stderr. The worker logs what + # matters through its own logger. + return + + def version_string(self) -> str: + # The base class names the exact Python version in the Server header. + return "aws-durable-execution-sdk-python-compute-worker" + + +class _Server(ThreadingHTTPServer): + """The worker's HTTP server.""" + + daemon_threads = True + # socketserver listens with a backlog of 5. A connection beyond the + # backlog can get a TCP reset. So the worker uses Node's backlog of 511, + # as the MicroVM worker does. The kernel caps it at net.core.somaxconn. + request_queue_size = 511 + + def __init__(self, address: tuple[str, int], runner: _JobRunner) -> None: + self.runner = runner + super().__init__(address, _RequestHandler) + + def server_bind(self) -> None: + # HTTPServer.server_bind looks up the host's fully qualified name, + # which can wait for DNS. The worker never uses the name. So the bind + # skips the lookup. + socketserver.TCPServer.server_bind(self) + self.server_name = "localhost" + self.server_port = int(self.server_address[1]) + + def handle_error(self, request: Any, client_address: Any) -> None: + # A client that resets the connection while it sends the request + # line or the headers ends here. It needs no traceback on stderr. + if isinstance(sys.exception(), ConnectionError): + return + super().handle_error(request, client_address) diff --git a/packages/aws-durable-execution-sdk-python-compute-worker/tests/agentcore/agentcore_worker_test.py b/packages/aws-durable-execution-sdk-python-compute-worker/tests/agentcore/agentcore_worker_test.py new file mode 100644 index 000000000..d6ecea7db --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-compute-worker/tests/agentcore/agentcore_worker_test.py @@ -0,0 +1,664 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""Tests for the AgentCore worker, through real HTTP requests on loopback.""" + +from __future__ import annotations + +import http.client +import inspect +import json +import threading +import time +from collections.abc import Callable, Iterator +from typing import Any + +import pytest + +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + AgentcoreJobContext, + AgentcoreWorker, + start_agentcore_worker, +) +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + worker as worker_module, +) +from aws_durable_execution_sdk_python_compute_worker.microvm.callback_reporter import ( + CallbackReporter, +) + + +WAIT = 10 + + +def job_body(callback_id: str = "cb-1", **fields: Any) -> dict[str, Any]: + return { + "version": 1, + "region": "us-west-2", + "callbackId": callback_id, + "input": {"n": 2}, + **fields, + } + + +def request( + worker: AgentcoreWorker, + method: str, + path: str, + body: Any = None, + headers: dict[str, str] | None = None, +) -> tuple[int, Any]: + connection = http.client.HTTPConnection("127.0.0.1", worker.port, timeout=WAIT) + try: + data = ( + None + if body is None + else (body if isinstance(body, bytes) else json.dumps(body).encode()) + ) + connection.request(method, path, body=data, headers=headers or {}) + response = connection.getresponse() + return response.status, json.loads(response.read() or b"null") + finally: + connection.close() + + +@pytest.fixture +def start( + fake_client: Any, logger: Any +) -> Iterator[Callable[..., tuple[AgentcoreWorker, Any, list[str]]]]: + workers: list[AgentcoreWorker] = [] + + def _start( + handler: Callable[[Any, AgentcoreJobContext], Any], client: Any = None + ) -> tuple[AgentcoreWorker, Any, list[str]]: + client = client or fake_client() + regions: list[str] = [] + + def create_client(region: str) -> Any: + regions.append(region) + return client + + worker = start_agentcore_worker( + handler=handler, + logger=logger, + create_client=create_client, + host="127.0.0.1", + port=0, + ) + workers.append(worker) + return worker, client, regions + + yield _start + for worker in workers: + worker.close() + + +@pytest.fixture +def clock(monkeypatch: pytest.MonkeyPatch) -> list[int]: + """Make each read of the worker's clock return the next second, from + 100. The list holds the values that the worker read.""" + reads: list[int] = [] + + def now() -> int: + reads.append(100 + len(reads)) + return reads[-1] + + monkeypatch.setattr(worker_module, "_now", now) + return reads + + +def ping(worker: AgentcoreWorker) -> Any: + status, answer = request(worker, "GET", "/ping") + assert status == 200 + return answer + + +def test_ping_keeps_the_start_time_while_the_status_stays( + start: Any, clock: list[int] +) -> None: + worker, _, _ = start(lambda job_input, context: None) + + first = ping(worker) + second = ping(worker) + + assert first == {"status": "Healthy", "time_of_last_update": 100} + assert second == first + assert clock == [100] + + +def test_ping_changes_the_time_when_the_first_job_starts_and_the_last_ends( + start: Any, clock: list[int] +) -> None: + release = threading.Event() + started = threading.Semaphore(0) + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + started.release() + assert release.wait(WAIT) + return "done" + + worker, client, _ = start(handler) + assert request(worker, "POST", "/invocations", job_body("cb-1")) == ( + 200, + {"accepted": True}, + ) + assert started.acquire(timeout=WAIT) + busy = ping(worker) + # A second job while one runs does not change the status. + request(worker, "POST", "/invocations", job_body("cb-2")) + assert started.acquire(timeout=WAIT) + still_busy = ping(worker) + release.set() + assert worker.idle(WAIT) + healthy = ping(worker) + + assert busy == {"status": "HealthyBusy", "time_of_last_update": 101} + assert still_busy == busy + assert healthy == {"status": "Healthy", "time_of_last_update": 102} + assert ping(worker) == healthy + assert client.names() == ["success", "success"] + + +def test_reports_the_result_to_the_callback_of_the_job(start: Any) -> None: + seen: list[AgentcoreJobContext] = [] + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + seen.append(context) + return {"sum": job_input["n"] + 1} + + worker, client, regions = start(handler) + status, _ = request( + worker, + "POST", + "/invocations", + job_body(), + {"X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": "s" * 36}, + ) + + assert status == 200 + assert worker.idle(WAIT) + assert client.calls == [("success", {"CallbackId": "cb-1", "Result": b'{"sum":3}'})] + assert regions == ["us-west-2"] + [context] = seen + assert context.callback_id == "cb-1" + assert context.session_id == "s" * 36 + assert context.region == "us-west-2" + + +def test_reports_the_error_of_a_failed_job(start: Any) -> None: + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + msg = "3 tests failed" + raise RuntimeError(msg) + + worker, client, _ = start(handler) + request(worker, "POST", "/invocations", job_body()) + + assert worker.idle(WAIT) + assert client.calls == [ + ( + "failure", + { + "CallbackId": "cb-1", + "Error": { + "ErrorType": "RuntimeError", + "ErrorMessage": "3 tests failed", + }, + }, + ) + ] + + +def test_reports_a_result_that_is_not_json_as_a_failure(start: Any) -> None: + worker, client, _ = start(lambda job_input, context: {1, 2}) + request(worker, "POST", "/invocations", job_body()) + + assert worker.idle(WAIT) + [(name, call)] = client.calls + assert name == "failure" + assert call["Error"]["ErrorType"] == "ResultSerializationError" + + +def test_runs_a_repeated_callback_id_once(start: Any) -> None: + runs: list[str] = [] + release = threading.Event() + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + runs.append(context.callback_id) + assert release.wait(WAIT) + return None + + worker, client, _ = start(handler) + # The second request arrives while the job runs, the third after it ended. + assert request(worker, "POST", "/invocations", job_body())[1] == {"accepted": True} + assert request(worker, "POST", "/invocations", job_body())[1] == {"accepted": True} + release.set() + assert worker.idle(WAIT) + assert request(worker, "POST", "/invocations", job_body())[1] == {"accepted": True} + assert worker.idle(WAIT) + + assert runs == ["cb-1"] + assert client.names() == ["success"] + + +def test_sends_heartbeats_for_a_job_with_a_heartbeat_timeout( + fake_client: Any, logger: Any +) -> None: + release = threading.Event() + client = fake_client() + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + assert release.wait(WAIT) + return None + + worker = start_agentcore_worker( + handler=handler, + logger=logger, + create_client=lambda region: client, + heartbeat_interval_seconds=0.05, + host="127.0.0.1", + port=0, + ) + try: + request(worker, "POST", "/invocations", job_body(heartbeatTimeoutSeconds=60)) + deadline = threading.Event() + for _ in range(200): + if client.names().count("heartbeat") >= 2: + break + deadline.wait(0.05) + release.set() + assert worker.idle(WAIT) + finally: + worker.close() + + names = client.names() + assert names.count("heartbeat") >= 2 + # Heartbeats continue while the worker reports the outcome. So a + # heartbeat can follow the success call. + assert names.count("success") == 1 + + +@pytest.mark.parametrize( + "body", + [ + b"not json", + job_body(version=2), + {"version": 1, "region": "us-west-2"}, + {"version": 1, "callbackId": "cb-1"}, + job_body(heartbeatTimeoutSeconds=0), + ], +) +def test_rejects_a_job_request_that_does_not_match_the_contract( + start: Any, body: Any +) -> None: + runs: list[Any] = [] + worker, client, _ = start(lambda job_input, context: runs.append(job_input)) + + status, answer = request(worker, "POST", "/invocations", body) + + assert status == 400 + assert "error" in answer + assert worker.idle(WAIT) + assert runs == [] + assert client.calls == [] + + +def test_answers_404_for_another_path(start: Any) -> None: + worker, _, _ = start(lambda job_input, context: None) + + assert request(worker, "GET", "/other")[0] == 404 + assert request(worker, "POST", "/ping", {})[0] == 404 + + +def test_refuses_an_async_handler() -> None: + async def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + return None + + with pytest.raises(TypeError, match="async"): + start_agentcore_worker(handler=handler, host="127.0.0.1", port=0) # type: ignore[arg-type] + + +def test_refuses_a_handler_that_is_not_callable() -> None: + with pytest.raises(TypeError, match="callable"): + start_agentcore_worker(handler="job", host="127.0.0.1", port=0) # type: ignore[arg-type] + + +def test_wait_returns_after_close(start: Any) -> None: + worker, _, _ = start(lambda job_input, context: None) + + assert not worker.wait(0) + worker.close() + worker.close() + assert worker.wait(0) + + +def test_refuses_an_object_whose_call_is_async() -> None: + class Handler: + async def __call__(self, job_input: Any, context: AgentcoreJobContext) -> Any: + return None + + with pytest.raises(TypeError, match="async"): + start_agentcore_worker(handler=Handler(), host="127.0.0.1", port=0) # type: ignore[arg-type] + + +def test_answers_500_and_forgets_a_job_whose_thread_did_not_start( + start: Any, logger: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + # The thread starts before the answer. So a failed start answers 500, + # and the worker forgets the job. A later delivery of the same job then + # runs it. + runs: list[Any] = [] + worker, client, _ = start(lambda job_input, context: runs.append(job_input)) + original = threading.Thread.start + + def fail_job_threads(self: threading.Thread) -> None: + if self.name.startswith("agentcore-job-"): + msg = "can't start new thread" + raise RuntimeError(msg) + original(self) + + monkeypatch.setattr(threading.Thread, "start", fail_job_threads) + assert request(worker, "POST", "/invocations", job_body())[0] == 500 + monkeypatch.setattr(threading.Thread, "start", original) + + assert ping(worker)["status"] == "Healthy" + assert "could not start the job's thread" in logger.messages("error") + assert runs == [] + assert client.calls == [] + + assert request(worker, "POST", "/invocations", job_body())[0] == 200 + assert worker.idle(WAIT) + assert runs == [{"n": 2}] + assert client.names() == ["success"] + + +def test_starts_the_handler_only_after_the_answer_is_written( + start: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + # The answer takes 0.3 seconds to write. A handler that started before + # the answer was written would record an earlier time. + events: list[str] = [] + original = worker_module._Exchange.respond # noqa: SLF001 + + def slow_respond(self: Any, status: int, body: Any) -> None: + if body == {"accepted": True}: + time.sleep(0.3) + original(self, status, body) + events.append("answered") + return + original(self, status, body) + + monkeypatch.setattr(worker_module._Exchange, "respond", slow_respond) # noqa: SLF001 + worker, _, _ = start(lambda job_input, context: events.append("handler")) + + assert request(worker, "POST", "/invocations", job_body())[0] == 200 + assert worker.idle(WAIT) + + assert events == ["answered", "handler"] + + +def test_forgets_the_oldest_finished_job_after_the_limit( + start: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(worker_module, "MAX_FINISHED_JOBS", 1) + runs: list[str] = [] + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + runs.append(context.callback_id) + + worker, _, _ = start(handler) + for callback_id in ("cb-1", "cb-2", "cb-1"): + request(worker, "POST", "/invocations", job_body(callback_id)) + assert worker.idle(WAIT) + + assert runs == ["cb-1", "cb-2", "cb-1"] + + +def test_logs_a_job_whose_client_cannot_be_created( + logger: Any, +) -> None: + def create_client(region: str) -> Any: + msg = "no credentials" + raise RuntimeError(msg) + + worker = start_agentcore_worker( + handler=lambda job_input, context: None, + logger=logger, + create_client=create_client, + host="127.0.0.1", + port=0, + ) + try: + assert request(worker, "POST", "/invocations", job_body())[0] == 200 + assert worker.idle(WAIT) + assert ping(worker)["status"] == "Healthy" + finally: + worker.close() + + assert logger.messages("error") == ["job failed inside the worker"] + + +def test_creates_its_own_reporter_without_create_client( + fake_client: Any, logger: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + client = fake_client() + regions: list[str] = [] + + def create(callback_id: str, region: str, *, logger: Any = None) -> Any: + regions.append(region) + return CallbackReporter(callback_id, client, logger=logger) + + monkeypatch.setattr(worker_module.CallbackReporter, "create", create) + worker = start_agentcore_worker( + handler=lambda job_input, context: "done", + logger=logger, + host="127.0.0.1", + port=0, + ) + try: + request(worker, "POST", "/invocations", job_body()) + assert worker.idle(WAIT) + finally: + worker.close() + + assert regions == ["us-west-2"] + assert client.names() == ["success"] + + +def test_does_not_report_a_job_whose_callback_is_gone( + fake_client: Any, make_error: Any, logger: Any +) -> None: + # The first heartbeat finds the callback closed. The worker then sets + # context.cancelled, and does not report the handler's outcome. + client = fake_client(make_error("CallbackTimeoutException")) + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + assert context.cancelled.wait(WAIT) + return "late" + + worker = start_agentcore_worker( + handler=handler, + logger=logger, + create_client=lambda region: client, + heartbeat_interval_seconds=0.05, + host="127.0.0.1", + port=0, + ) + try: + request(worker, "POST", "/invocations", job_body(heartbeatTimeoutSeconds=60)) + assert worker.idle(WAIT) + finally: + worker.close() + + assert client.names() == ["heartbeat"] + assert "job outcome not reported: the callback is gone" in logger.messages( + "warning" + ) + + +def test_logs_an_outcome_that_could_not_be_reported( + fake_client: Any, make_error: Any, logger: Any +) -> None: + client = fake_client(make_error("AccessDeniedException", 403)) + worker = start_agentcore_worker( + handler=lambda job_input, context: "done", + logger=logger, + create_client=lambda region: client, + host="127.0.0.1", + port=0, + ) + try: + request(worker, "POST", "/invocations", job_body()) + assert worker.idle(WAIT) + finally: + worker.close() + + assert client.names() == ["success"] + assert logger.messages("error") == ["job outcome could not be reported"] + + +def test_the_contract_constants_are_the_agentcore_host_and_port() -> None: + assert worker_module.AGENTCORE_HOST == "0.0.0.0" # noqa: S104 + assert worker_module.AGENTCORE_PORT == 8080 + parameters = inspect.signature(start_agentcore_worker).parameters + assert parameters["port"].default == 8080 + assert parameters["host"].default is None + + +def test_default_host_is_loopback_outside_agentcore( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.delenv("AGENTCORE_RUNTIME_URL", raising=False) + + assert worker_module.default_host() == "127.0.0.1" + + +def test_default_host_is_all_interfaces_inside_agentcore( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("AGENTCORE_RUNTIME_URL", "http://127.0.0.1:9000") + + assert worker_module.default_host() == "0.0.0.0" # noqa: S104 + + +def test_binds_loopback_without_a_host_outside_agentcore( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.delenv("AGENTCORE_RUNTIME_URL", raising=False) + worker = start_agentcore_worker(handler=lambda job_input, context: None, port=0) + try: + assert worker._server.server_address[0] == "127.0.0.1" # noqa: SLF001 + assert ping(worker)["status"] == "Healthy" + finally: + worker.close() + + +def test_binds_the_explicit_host_inside_agentcore( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("AGENTCORE_RUNTIME_URL", "http://127.0.0.1:9000") + worker = start_agentcore_worker( + handler=lambda job_input, context: None, host="127.0.0.1", port=0 + ) + try: + assert worker._server.server_address[0] == "127.0.0.1" # noqa: SLF001 + finally: + worker.close() + + +def test_listens_with_a_backlog_of_511(start: Any) -> None: + worker, _, _ = start(lambda job_input, context: None) + + assert worker._server.request_queue_size == 511 # noqa: SLF001 + + +def test_handle_error_prints_no_traceback_for_a_reset_connection( + start: Any, capsys: pytest.CaptureFixture[str] +) -> None: + worker, _, _ = start(lambda job_input, context: None) + server = worker._server # noqa: SLF001 + + try: + raise ConnectionResetError + except ConnectionError: + server.handle_error(None, ("127.0.0.1", 1)) + assert capsys.readouterr().err == "" + try: + raise ValueError("boom") # noqa: EM101, TRY003 + except ValueError: + server.handle_error(None, ("127.0.0.1", 1)) + assert "ValueError: boom" in capsys.readouterr().err + + +def test_fails_the_callback_when_the_heartbeats_cannot_start( + start: Any, logger: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + runs: list[Any] = [] + + def fail_start(*args: Any, **kwargs: Any) -> Any: + msg = "can't start new thread" + raise RuntimeError(msg) + + monkeypatch.setattr(worker_module.Heartbeats, "start", fail_start) + worker, client, _ = start(lambda job_input, context: runs.append(job_input)) + + assert request(worker, "POST", "/invocations", job_body())[1] == {"accepted": True} + assert worker.idle(WAIT) + assert runs == [] + assert client.calls == [ + ( + "failure", + { + "CallbackId": "cb-1", + "Error": { + "ErrorType": "RuntimeError", + "ErrorMessage": "can't start new thread", + }, + }, + ) + ] + assert logger.messages("error") == ["could not start the job's heartbeats"] + + +class _SlowSet(set[str]): + """A set whose membership check reads the set, then waits 0.2 seconds. + + Two requests that check without a lock both read the set before either + adds the callback ID. So both would run the job. The worker's lock makes + the second check wait for the first add. + """ + + def __contains__(self, item: object) -> bool: + found = super().__contains__(item) + time.sleep(0.2) + return found + + +def test_two_concurrent_deliveries_of_one_job_run_it_once(start: Any) -> None: + runs: list[str] = [] + release = threading.Event() + + def handler(job_input: Any, context: AgentcoreJobContext) -> Any: + runs.append(context.callback_id) + assert release.wait(WAIT) + + worker, client, _ = start(handler) + worker._runner._running = _SlowSet() # noqa: SLF001 + answers: list[Any] = [] + threads = [ + threading.Thread( + target=lambda: answers.append( + request(worker, "POST", "/invocations", job_body()) + ) + ) + for _ in range(2) + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join(WAIT) + release.set() + assert worker.idle(WAIT) + + assert answers == [(200, {"accepted": True})] * 2 + assert runs == ["cb-1"] + assert client.names() == ["success"] diff --git a/packages/aws-durable-execution-sdk-python-examples/agentcore-agent/app.py b/packages/aws-durable-execution-sdk-python-examples/agentcore-agent/app.py new file mode 100644 index 000000000..e0dcebd1a --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-examples/agentcore-agent/app.py @@ -0,0 +1,30 @@ +"""The agent that the AgentCore example runs in its agent runtime. + +Deploy it as the entry point of an AgentCore Runtime agent runtime, with the +compute worker package installed. The agent runtime's execution role needs +lambda:SendDurableExecutionCallbackSuccess, +lambda:SendDurableExecutionCallbackFailure, and +lambda:SendDurableExecutionCallbackHeartbeat on the example function. +""" + +import hashlib +from typing import Any + +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + AgentcoreJobContext, + start_agentcore_worker, +) + + +def count_words(job_input: dict[str, Any], context: AgentcoreJobContext) -> dict: + """Count the words of the text, and hash it.""" + text: str = job_input["text"] + return { + "wordCount": len(text.split()), + "sha256": hashlib.sha256(text.encode()).hexdigest(), + "sessionId": context.session_id, + } + + +if __name__ == "__main__": + start_agentcore_worker(handler=count_words).wait() diff --git a/packages/aws-durable-execution-sdk-python-examples/examples-catalog.json b/packages/aws-durable-execution-sdk-python-examples/examples-catalog.json index 8a8169353..b1811fa6d 100644 --- a/packages/aws-durable-execution-sdk-python-examples/examples-catalog.json +++ b/packages/aws-durable-execution-sdk-python-examples/examples-catalog.json @@ -847,6 +847,18 @@ "ExecutionTimeout": 900 }, "path": "./src/microvm/microvm_session_suspend_after.py" + }, + { + "name": "AgentCore Run Job", + "description": "Runs one job in an AgentCore Runtime session and returns the result that the agent reports", + "handler": "agentcore_run_job.handler", + "integration": true, + "usesAgentcore": true, + "durableConfig": { + "RetentionPeriodInDays": 7, + "ExecutionTimeout": 900 + }, + "path": "./src/agentcore/agentcore_run_job.py" } ] } diff --git a/packages/aws-durable-execution-sdk-python-examples/scripts/build_examples.py b/packages/aws-durable-execution-sdk-python-examples/scripts/build_examples.py index b30450a63..4ab5fb46f 100644 --- a/packages/aws-durable-execution-sdk-python-examples/scripts/build_examples.py +++ b/packages/aws-durable-execution-sdk-python-examples/scripts/build_examples.py @@ -20,8 +20,9 @@ def build_examples() -> None: runtime_packages = [ packages_dir / "aws-durable-execution-sdk-python", - # The MicroVM examples import the extras. The examples package does - # not depend on the extras, so pip installs the extras only from here. + # The MicroVM and AgentCore examples import the extras. The examples + # package does not depend on the extras, so pip installs the extras + # only from here. packages_dir / "aws-durable-execution-sdk-python-extras", packages_dir / "aws-durable-execution-sdk-python-otel", packages_dir / "aws-durable-execution-sdk-python-testing", diff --git a/packages/aws-durable-execution-sdk-python-examples/scripts/generate_function_name_map.py b/packages/aws-durable-execution-sdk-python-examples/scripts/generate_function_name_map.py index 513c137cd..83f30180c 100644 --- a/packages/aws-durable-execution-sdk-python-examples/scripts/generate_function_name_map.py +++ b/packages/aws-durable-execution-sdk-python-examples/scripts/generate_function_name_map.py @@ -4,35 +4,49 @@ import json from typing import Any -from generate_sam_template import load_catalog, to_logical_id, uses_microvm +from generate_sam_template import ( + load_catalog, + to_logical_id, + uses_agentcore, + uses_microvm, +) def build_function_name_map( examples: list[dict[str, Any]], function_name_prefix: str, include_microvm: bool = False, + include_agentcore: bool = False, ) -> dict[str, str]: """Map each example name to its qualified Lambda function name. The stack deploys a MicroVM example only when CI passes an image. So the map leaves a MicroVM example out unless ``include_microvm`` is true. The - conftest skips a test whose example is missing from the map. + conftest skips a test whose example is missing from the map. The same + holds for an AgentCore example, an agent runtime, and + ``include_agentcore``. """ return { example["name"]: ( f"{function_name_prefix}{to_logical_id(example['handler'])}:$LATEST" ) for example in examples - if include_microvm or not uses_microvm(example) + if (include_microvm or not uses_microvm(example)) + and (include_agentcore or not uses_agentcore(example)) } def generate_function_name_map( - function_name_prefix: str, include_microvm: bool = False + function_name_prefix: str, + include_microvm: bool = False, + include_agentcore: bool = False, ) -> dict[str, str]: """Generate a test-name to qualified Lambda function-name map.""" return build_function_name_map( - load_catalog()["examples"], function_name_prefix, include_microvm + load_catalog()["examples"], + function_name_prefix, + include_microvm, + include_agentcore, ) @@ -53,12 +67,22 @@ def main() -> int: "the stack was deployed with MicrovmImageArn." ), ) + parser.add_argument( + "--include-agentcore", + action="store_true", + help=( + 'Include the examples with "usesAgentcore": true. Pass it only ' + "when the stack was deployed with AgentRuntimeArn." + ), + ) args = parser.parse_args() print( json.dumps( generate_function_name_map( - args.function_name_prefix, include_microvm=args.include_microvm + args.function_name_prefix, + include_microvm=args.include_microvm, + include_agentcore=args.include_agentcore, ), separators=(",", ":"), ) diff --git a/packages/aws-durable-execution-sdk-python-examples/scripts/generate_sam_template.py b/packages/aws-durable-execution-sdk-python-examples/scripts/generate_sam_template.py index 3c1e63ffe..5af380358 100644 --- a/packages/aws-durable-execution-sdk-python-examples/scripts/generate_sam_template.py +++ b/packages/aws-durable-execution-sdk-python-examples/scripts/generate_sam_template.py @@ -10,6 +10,7 @@ DEFAULT_DURABLE_LOGGING_CONFIG: dict[str, str] = {"LogFormat": "JSON"} LOG_RETENTION_DAYS = 7 MICROVM_CONDITION = "HasMicrovmImage" +AGENTCORE_CONDITION = "HasAgentRuntime" def uses_microvm(example: dict[str, Any]) -> bool: @@ -17,6 +18,11 @@ def uses_microvm(example: dict[str, Any]) -> bool: return example.get("usesMicrovm") is True +def uses_agentcore(example: dict[str, Any]) -> bool: + """Return true when the catalog example runs a job in AgentCore Runtime.""" + return example.get("usesAgentcore") is True + + def load_catalog() -> dict[str, Any]: """Load the examples catalog.""" catalog_path = Path(__file__).resolve().parent.parent / "examples-catalog.json" @@ -92,6 +98,15 @@ def build_template(examples: list[dict[str, Any]]) -> dict[str, Any]: "examples run with." ), }, + "AgentRuntimeArn": { + "Type": "String", + "Default": "", + "Description": ( + "ARN of the AgentCore Runtime agent runtime for the " + "AgentCore examples. Empty leaves the AgentCore examples " + "out of the stack." + ), + }, }, "Resources": {}, } @@ -101,19 +116,23 @@ def build_template(examples: list[dict[str, Any]]) -> dict[str, Any]: # function and its log group exist only when both parameters are set. # CloudFormation linters flag an unused condition. So the template declares # the condition only when an example uses it. + conditions: dict[str, Any] = {} if any(uses_microvm(example) for example in examples): - template["Conditions"] = { - MICROVM_CONDITION: { - "Fn::And": [ - {"Fn::Not": [{"Fn::Equals": [{"Ref": "MicrovmImageArn"}, ""]}]}, - { - "Fn::Not": [ - {"Fn::Equals": [{"Ref": "MicrovmExecutionRoleArn"}, ""]} - ] - }, - ] - } + conditions[MICROVM_CONDITION] = { + "Fn::And": [ + {"Fn::Not": [{"Fn::Equals": [{"Ref": "MicrovmImageArn"}, ""]}]}, + {"Fn::Not": [{"Fn::Equals": [{"Ref": "MicrovmExecutionRoleArn"}, ""]}]}, + ] + } + # An AgentCore example sends its job to the agent runtime named by + # AgentRuntimeArn. So, as for MicroVM, its function and log group exist + # only when the parameter is set. + if any(uses_agentcore(example) for example in examples): + conditions[AGENTCORE_CONDITION] = { + "Fn::Not": [{"Fn::Equals": [{"Ref": "AgentRuntimeArn"}, ""]}] } + if conditions: + template["Conditions"] = conditions for example in examples: logical_id = to_logical_id(example["handler"]) @@ -151,6 +170,8 @@ def build_template(examples: list[dict[str, Any]]) -> dict[str, Any]: # SAM merges these variables with the Globals variables. variables["MICROVM_IMAGE_ARN"] = {"Ref": "MicrovmImageArn"} variables["MICROVM_EXECUTION_ROLE_ARN"] = {"Ref": "MicrovmExecutionRoleArn"} + if uses_agentcore(example): + variables["AGENT_RUNTIME_ARN"] = {"Ref": "AgentRuntimeArn"} if variables: properties["Environment"] = {"Variables": variables} @@ -173,6 +194,9 @@ def build_template(examples: list[dict[str, Any]]) -> dict[str, Any]: # generates for the function. function_resource["Condition"] = MICROVM_CONDITION log_group_resource["Condition"] = MICROVM_CONDITION + if uses_agentcore(example): + function_resource["Condition"] = AGENTCORE_CONDITION + log_group_resource["Condition"] = AGENTCORE_CONDITION template["Resources"][logical_id] = function_resource template["Resources"][f"{logical_id}LogGroup"] = log_group_resource diff --git a/packages/aws-durable-execution-sdk-python-examples/src/agentcore/agentcore_run_job.py b/packages/aws-durable-execution-sdk-python-examples/src/agentcore/agentcore_run_job.py new file mode 100644 index 000000000..4692d7d23 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-examples/src/agentcore/agentcore_run_job.py @@ -0,0 +1,52 @@ +"""Run one job in an AgentCore Runtime session and return its result. + +The agent runtime runs agentcore-agent/app.py. Its handler counts the words +of the text and hashes it. +""" + +import os +from typing import Any + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import DurableContext +from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python_extras.compute.agentcore import ( + AgentcoreConfig, + agentcore, +) + + +def _require_agent_runtime_arn() -> str: + """Return the agent runtime ARN. + + The SAM template sets ``AGENT_RUNTIME_ARN`` on every example with + ``"usesAgentcore": true``, from the ``AgentRuntimeArn`` parameter. + """ + arn = os.environ.get("AGENT_RUNTIME_ARN") + if not arn: + msg = ( + "AGENT_RUNTIME_ARN must be set. Deploy an agent runtime that runs " + "agentcore-agent/app.py, and pass its ARN as AgentRuntimeArn." + ) + raise RuntimeError(msg) + return arn + + +@durable_execution +def handler(event: dict[str, Any], context: DurableContext) -> Any: + # The operation starts a new session with the job, and waits on a durable + # callback. The worker in the session runs the job and completes the + # callback. The operation then stops the session. + return agentcore( + context, + "count-words", + job_input={"text": event["text"]}, + config=AgentcoreConfig( + agent_runtime_arn=_require_agent_runtime_arn(), + qualifier="DEFAULT", + timeout=Duration.from_minutes(5), + # A session that fails to start fails the operation after this + # time, instead of after the timeout. + heartbeat_timeout=Duration.from_seconds(60), + ), + ) diff --git a/packages/aws-durable-execution-sdk-python-examples/test/README.md b/packages/aws-durable-execution-sdk-python-examples/test/README.md index cce0928a7..56676b4d1 100644 --- a/packages/aws-durable-execution-sdk-python-examples/test/README.md +++ b/packages/aws-durable-execution-sdk-python-examples/test/README.md @@ -139,6 +139,16 @@ def test_my_example(durable_runner): - `usesMicrovm` - `true` marks an example that launches a MicroVM. The template then gives its function and log group the condition `HasMicrovmImage`. The condition holds only when the parameters `MicrovmImageArn` and `MicrovmExecutionRoleArn` are both set. The function receives them as `MICROVM_IMAGE_ARN` and `MICROVM_EXECUTION_ROLE_ARN`. So a deploy without an image leaves the example out of the stack. `scripts/generate_function_name_map.py` leaves out the `usesMicrovm` examples unless you pass `--include-microvm`. Pass it only when the stack was deployed with an image. A test whose example is missing from the map is skipped. +- `usesAgentcore` - `true` marks an example that runs a job in an Amazon Bedrock AgentCore Runtime session. The template then gives its function and log group the condition `HasAgentRuntime`. The condition holds only when the parameter `AgentRuntimeArn` is set. The function receives it as `AGENT_RUNTIME_ARN`. So a deploy without an agent runtime leaves the example out of the stack. `scripts/generate_function_name_map.py` leaves the example out unless you pass `--include-agentcore`. + +### AgentCore Example and Test +`test/agentcore/test_agentcore_run_job.py` runs the `AgentCore Run Job` example end to end: `InvokeAgentRuntime`, the session, the worker, the callback, and `StopRuntimeSession`. It needs these resources, which CI does not create yet: + +1. An agent runtime whose entry point is `agentcore-agent/app.py`, with the compute worker package installed. +2. On the agent runtime's execution role: `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the example function. +3. On the function's role (`LambdaExecutionRoleArn`): `bedrock-agentcore:InvokeAgentRuntime` and `bedrock-agentcore:StopRuntimeSession` on the agent runtime ARN and on `/runtime-endpoint/*`. + +To run it, pass the agent runtime ARN to `sam deploy` as `AgentRuntimeArn`, and generate the function-name map with `--include-agentcore`. Without both, the test is skipped. ### MicroVM Image `microvm-image/` holds the app, the Dockerfile, and the dependency pins of the MicroVM image that the MicroVM examples run. `hatch run examples:ensure-microvm-image --output image.json` builds the image or reuses it. It needs `TEST_ACCOUNT_ID`, `AWS_REGION`, and AWS credentials for the test account. diff --git a/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_example_parts.py b/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_example_parts.py new file mode 100644 index 000000000..b8021e38c --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_example_parts.py @@ -0,0 +1,73 @@ +"""Local tests for the parts of the AgentCore example. + +The cloud test needs a deployed agent runtime. These tests check the +example's settings, its catalog entry, and the agent's handler without one. +""" + +import hashlib +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + + +# The example imports the extras. The examples package does not depend on +# the extras. So an environment without them skips this module. +pytest.importorskip("aws_durable_execution_sdk_python_extras") + +from src.agentcore import agentcore_run_job # noqa: E402 + + +PACKAGE_DIR = Path(__file__).resolve().parents[2] +RUNTIME_ARN = "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/agent-abc" + + +def test_returns_the_agent_runtime_arn(monkeypatch): + monkeypatch.setenv("AGENT_RUNTIME_ARN", RUNTIME_ARN) + + assert agentcore_run_job._require_agent_runtime_arn() == RUNTIME_ARN + + +@pytest.mark.parametrize("unset", [True, False], ids=["unset", "empty"]) +def test_names_the_variable_when_it_is_missing(unset, monkeypatch): + if unset: + monkeypatch.delenv("AGENT_RUNTIME_ARN", raising=False) + else: + monkeypatch.setenv("AGENT_RUNTIME_ARN", "") + + with pytest.raises(RuntimeError, match="AGENT_RUNTIME_ARN must be set"): + agentcore_run_job._require_agent_runtime_arn() + + +def test_the_catalog_entry_needs_an_agent_runtime(): + examples = json.loads( + (PACKAGE_DIR / "examples-catalog.json").read_text(encoding="utf-8") + )["examples"] + entry = next(e for e in examples if e["name"] == "AgentCore Run Job") + + assert entry["usesAgentcore"] is True + assert entry["handler"] == "agentcore_run_job.handler" + assert (PACKAGE_DIR / entry["path"]).is_file() + + +def test_the_agent_counts_the_words_and_returns_the_session_id(): + # The agent imports the worker, which the dev-examples environment does + # not install. So that environment skips this test. + pytest.importorskip("aws_durable_execution_sdk_python_compute_worker.agentcore") + path = PACKAGE_DIR / "agentcore-agent" / "app.py" + spec = importlib.util.spec_from_file_location("agentcore_agent_app", path) + assert spec is not None + assert spec.loader is not None + app = importlib.util.module_from_spec(spec) + spec.loader.exec_module(app) + text = "the quick brown fox\njumps" + + output = app.count_words({"text": text}, SimpleNamespace(session_id="s-1")) + + assert output == { + "wordCount": 5, + "sha256": hashlib.sha256(text.encode()).hexdigest(), + "sessionId": "s-1", + } diff --git a/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_run_job.py b/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_run_job.py new file mode 100644 index 000000000..54483b0bb --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-examples/test/agentcore/test_agentcore_run_job.py @@ -0,0 +1,103 @@ +"""Cloud test for the AgentCore run-job example. + +The test runs the real path: the durable function, InvokeAgentRuntime, the +AgentCore session, the worker, the callback, and StopRuntimeSession. The +local runner cannot start an AgentCore session. So the test is cloud-only. + +The cloud run needs a deployed agent runtime that runs agentcore-agent/app.py. +The stack deploys the example only when AgentRuntimeArn is set. Without it, +the function-name map leaves the example out, and the test is skipped. +""" + +import hashlib + +import pytest +from aws_durable_execution_sdk_python.execution import InvocationStatus +from aws_durable_execution_sdk_python.lambda_service import ( + OperationStatus, + OperationType, +) + + +# The examples package does not depend on the extras. So an environment +# without the extras skips this module instead of failing to import the +# example. +pytest.importorskip("aws_durable_execution_sdk_python_extras") + +from src.agentcore import agentcore_run_job # noqa: E402 +from test.conftest import deserialize_operation_payload # noqa: E402 + + +TEXT = "the quick brown fox jumps over the lazy dog" + + +@pytest.mark.example +@pytest.mark.durable_execution( + handler=agentcore_run_job.handler, + lambda_function_name="AgentCore Run Job", + cloud_only=True, +) +def test_runs_the_job_in_an_agentcore_session_and_returns_its_result( + durable_runner, +): + with durable_runner: + # A synchronous Invoke returns only when the execution ends. The cloud + # runner's boto3 client reads for at most 60 seconds, and it sends the + # Invoke again after a read timeout. Each resend starts a new + # execution. So the test starts the execution asynchronously and + # polls for its result. + execution_arn = durable_runner.run_async(input={"text": TEXT}, timeout=300) + result = durable_runner.wait_for_result( + execution_arn=execution_arn, timeout=300 + ) + + assert result.status is InvocationStatus.SUCCEEDED, result.error + output = deserialize_operation_payload(result.result) + assert output["wordCount"] == 9 + assert output["sha256"] == hashlib.sha256(TEXT.encode()).hexdigest() + + # The operation runs its durable operations in one child context named + # after the call. + context = result.get_context("count-words") + assert context.operation_type is OperationType.CONTEXT + assert context.status is OperationStatus.SUCCEEDED + assert context.sub_type == "Agentcore" + assert [ + (child.name, child.operation_type, child.status, child.sub_type) + for child in context.child_operations + ] == [ + ( + "count-words.session", + OperationType.STEP, + OperationStatus.SUCCEEDED, + "AgentcoreSessionId", + ), + ( + "count-words.callback", + OperationType.CALLBACK, + OperationStatus.SUCCEEDED, + "AgentcoreCallback", + ), + ( + "count-words.request", + OperationType.STEP, + OperationStatus.SUCCEEDED, + "AgentcoreRequest", + ), + ( + "count-words.stop", + OperationType.STEP, + OperationStatus.SUCCEEDED, + "AgentcoreStop", + ), + ] + + # AgentCore passed the session ID to the worker in a request header. So + # the job's session is the one that the session step checkpointed. + session_id = deserialize_operation_payload( + context.get_step("count-words.session").result + ) + assert output["sessionId"] == session_id + # A failed StopRuntimeSession is logged, not raised. So the check reads + # the stop step's own outcome. + assert context.get_step("count-words.stop").error is None diff --git a/packages/aws-durable-execution-sdk-python-examples/test/test_generate_function_name_map.py b/packages/aws-durable-execution-sdk-python-examples/test/test_generate_function_name_map.py index b9d8153b1..7e6dc9b4f 100644 --- a/packages/aws-durable-execution-sdk-python-examples/test/test_generate_function_name_map.py +++ b/packages/aws-durable-execution-sdk-python-examples/test/test_generate_function_name_map.py @@ -26,6 +26,11 @@ "handler": "microvm_run_job.handler", "usesMicrovm": True, }, + { + "name": "AgentCore Run Job", + "handler": "agentcore_run_job.handler", + "usesAgentcore": True, + }, ] @@ -44,12 +49,22 @@ def test_map_includes_microvm_examples_on_request(): } -def test_map_covers_every_catalog_example_without_microvm(): +def test_map_includes_agentcore_examples_on_request(): + assert generate_function_name_map.build_function_name_map( + EXAMPLES, "Py313-", include_agentcore=True + ) == { + "Hello World": "Py313-HelloWorld:$LATEST", + "AgentCore Run Job": "Py313-AgentcoreRunJob:$LATEST", + } + + +def test_map_covers_every_catalog_example_without_microvm_or_agentcore(): catalog = generate_function_name_map.load_catalog()["examples"] expected = { example["name"] for example in catalog if not generate_function_name_map.uses_microvm(example) + and not generate_function_name_map.uses_agentcore(example) } assert set(generate_function_name_map.generate_function_name_map("P-")) == ( diff --git a/packages/aws-durable-execution-sdk-python-examples/test/test_generate_sam_template.py b/packages/aws-durable-execution-sdk-python-examples/test/test_generate_sam_template.py index 05016e875..a1a279859 100644 --- a/packages/aws-durable-execution-sdk-python-examples/test/test_generate_sam_template.py +++ b/packages/aws-durable-execution-sdk-python-examples/test/test_generate_sam_template.py @@ -63,6 +63,13 @@ def test_build_template_adds_default_json_logging_to_durable_functions(): "usesMicrovm": True, "durableConfig": {"RetentionPeriodInDays": 7, "ExecutionTimeout": 900}, } +AGENTCORE_EXAMPLE = { + "name": "AgentCore Example", + "handler": "agentcore_example.handler", + "description": "An AgentCore example", + "usesAgentcore": True, + "durableConfig": {"RetentionPeriodInDays": 7, "ExecutionTimeout": 900}, +} PLAIN_EXAMPLE = { "name": "Hello World", "handler": "hello_world.handler", @@ -141,3 +148,35 @@ def test_build_template_sets_timeout_from_catalog(): assert template["Resources"]["HelloWorld"]["Properties"]["Timeout"] == 120 assert template["Globals"]["Function"]["Timeout"] == 60 + + +def test_build_template_declares_the_agent_runtime_parameter_empty(): + template = generate_sam_template.build_template([PLAIN_EXAMPLE]) + + parameter = template["Parameters"]["AgentRuntimeArn"] + assert parameter["Default"] == "" + assert parameter["Type"] == "String" + + +def test_build_template_requires_the_agent_runtime_for_agentcore_examples(): + template = generate_sam_template.build_template( + [PLAIN_EXAMPLE, MICROVM_EXAMPLE, AGENTCORE_EXAMPLE] + ) + + assert template["Conditions"]["HasAgentRuntime"] == { + "Fn::Not": [{"Fn::Equals": [{"Ref": "AgentRuntimeArn"}, ""]}] + } + assert "HasMicrovmImage" in template["Conditions"] + resources = template["Resources"] + assert resources["AgentcoreExample"]["Condition"] == "HasAgentRuntime" + assert resources["AgentcoreExampleLogGroup"]["Condition"] == "HasAgentRuntime" + assert resources["AgentcoreExample"]["Properties"]["Environment"] == { + "Variables": {"AGENT_RUNTIME_ARN": {"Ref": "AgentRuntimeArn"}} + } + assert "Condition" not in resources["HelloWorld"] + + +def test_build_template_declares_only_the_agentcore_condition(): + template = generate_sam_template.build_template([AGENTCORE_EXAMPLE]) + + assert set(template["Conditions"]) == {"HasAgentRuntime"} diff --git a/packages/aws-durable-execution-sdk-python-extras/README.md b/packages/aws-durable-execution-sdk-python-extras/README.md index 138215e54..7aa773d7f 100644 --- a/packages/aws-durable-execution-sdk-python-extras/README.md +++ b/packages/aws-durable-execution-sdk-python-extras/README.md @@ -2,7 +2,7 @@ > This package is experimental. Its API can change in any release. -Run jobs from a durable function in AWS Lambda MicroVMs, with the `microvm` and `microvm_session` operations. Ask a decision model or a large language model in a durable step, with the `decider` and `llm` operations ([docs/ai.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-extras/docs/ai.md)). +Run jobs from a durable function in AWS Lambda MicroVMs, with the `microvm` and `microvm_session` operations. Run a job in an Amazon Bedrock AgentCore Runtime session with the `agentcore` operation. Ask a decision model or a large language model in a durable step, with the `decider` and `llm` operations ([docs/ai.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-extras/docs/ai.md)). ## Install @@ -221,6 +221,82 @@ The function's role does not need `lambda:ResumeMicrovm`. The MicroVM's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the function ARN. With `auto_suspend_on_idle`, it also needs `lambda:SuspendMicrovm` on the image. +## Run a job in AgentCore Runtime + +> The `compute.agentcore` module is experimental. Its API can change in any release. + +`agentcore` sends one job to a new [Amazon Bedrock AgentCore Runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agents-tools-runtime.html) session, waits for the result, and stops the session. The function invocation ends while the agent works. So no Lambda compute is billed during the wait. + +```python +import os + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import DurableContext +from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python_extras.compute.agentcore import ( + AgentcoreConfig, + agentcore, +) + +AGENT = AgentcoreConfig( + agent_runtime_arn=os.environ["AGENT_RUNTIME_ARN"], + qualifier="DEFAULT", + timeout=Duration.from_hours(1), + heartbeat_timeout=Duration.from_minutes(5), +) + + +@durable_execution +def handler(event: dict, context: DurableContext) -> bool: + review = agentcore(context, "review", job_input={"pr": event["pr"]}, config=AGENT) + return review["approved"] +``` + +The agent runtime runs the [AgentCore worker](https://github.com/aws/aws-durable-execution-sdk-python/tree/main/packages/aws-durable-execution-sdk-python-compute-worker#run-a-job-in-agentcore-runtime): + +```python +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + AgentcoreJobContext, + start_agentcore_worker, +) + + +def review(job_input: dict, context: AgentcoreJobContext) -> dict: + return {"approved": True, "comments": []} + + +if __name__ == "__main__": + start_agentcore_worker(handler=review).wait() +``` + +The operation records `review.session`, `review.callback`, `review.request`, and `review.stop` in one child context named `review`. Each operation records a subtype from `AgentcoreOperationSubType`: `Agentcore` on the child context, then `AgentcoreSessionId`, `AgentcoreCallback`, `AgentcoreRequest`, and `AgentcoreStop`, the same values as the JavaScript SDK. Execution history and plugins can filter on them. `review.stop` calls `StopRuntimeSession` after success, failure, and timeout alike. A failed stop is logged, and does not hide the job's result or error. + +Each `InvokeAgentRuntime` and `StopRuntimeSession` call must end 10 seconds before the Lambda invocation times out, as the MicroVM calls must. A Lambda timeout during a step attempt records no outcome for the attempt. So a call that has not ended by then fails its attempt with a `TimeoutError`, and the default retry strategy retries it in a later invocation. A step attempt with less than 11 seconds left makes no call. It fails with a message that asks for a longer function timeout. + +While a job runs, the worker answers `/ping` with `HealthyBusy`, so the agent runtime's idle timeout (`idleRuntimeSessionTimeout`) does not end the job. After the idle timeout, a session with no running job ends. A later request with the same session ID then gets a new microVM without an error, and the memory and files of the old one are lost. The `agentcore` operation sends one job per session, so this loss cannot affect a job. + +Delivery is at-least-once, so make the job handler's side effects idempotent: + +1. `InvokeAgentRuntime` has no idempotency token. So a retried request step can deliver the same job twice. +2. The worker ignores a second delivery of a job that it already accepted. It remembers the callback IDs only in its process memory. So the worker runs a job once per process. +3. An answer can be lost after the job finished. The retry can then arrive after the idle timeout replaced the session's process. The new process runs the same job again. + +### Rules for AgentCore jobs + +1. Set `heartbeat_timeout`. It ends the job when the worker crashes or stops reporting. The first heartbeat period starts when the callback is created, before the job is delivered. So it must cover the request step with its retries and a cold session start, which took 1.4 to 3.1 seconds in a live test. 30 seconds or more is a safe value. +2. `timeout` must fit the agent runtime's configured `maxLifetime`. +3. A job can run more than once when AgentCore replaces the worker process. Make handler side effects safe to repeat. +4. Your logger must not throw. +5. The runtime's idle timeout applies only while no job is running. One `agentcore()` call runs one job, so it does not lose state between jobs. +6. Give the durable function a timeout of at least 11 seconds. Each AgentCore call needs the 10-second invocation reserve and 1 second for the call. + +Every AgentCore failure is an `AgentcoreError`, and every `AgentcoreError` is a `ComputeError`. The subclasses are `AgentcoreDeliveryError`, `AgentcoreJobFailedError`, and `AgentcoreTimeoutError`. + +Permissions: + +- The function's execution role needs `bedrock-agentcore:InvokeAgentRuntime` and `bedrock-agentcore:StopRuntimeSession` on two resources: the agent runtime ARN, and `/runtime-endpoint/*`. Without the endpoint resource, both calls are denied on `/runtime-endpoint/DEFAULT`. To allow one endpoint only, name the endpoint of `qualifier` instead of `*`. +- The agent runtime's execution role needs `lambda:SendDurableExecutionCallbackSuccess`, `lambda:SendDurableExecutionCallbackFailure`, and `lambda:SendDurableExecutionCallbackHeartbeat` on the function ARN. + ## Reference - [docs/microvm.md](https://github.com/aws/aws-durable-execution-sdk-python/blob/main/packages/aws-durable-execution-sdk-python-extras/docs/microvm.md): job delivery, the durable operations each call records, all errors, the retry tiers, the state check, every permission, and best practices. diff --git a/packages/aws-durable-execution-sdk-python-extras/pyproject.toml b/packages/aws-durable-execution-sdk-python-extras/pyproject.toml index e7630e835..55a2cb796 100644 --- a/packages/aws-durable-execution-sdk-python-extras/pyproject.toml +++ b/packages/aws-durable-execution-sdk-python-extras/pyproject.toml @@ -21,12 +21,16 @@ classifiers = [ "Programming Language :: Python :: Implementation :: CPython", ] dependencies = [ - # 2.0.0 is the first release with the graded callback errors + # 3.0.0 is the first release with custom subtypes (the sub_type field of + # StepConfig, CallbackConfig, and ChildConfig), which the agentcore + # operation records. 2.0.0 added the graded callback errors # (CallbackTimeoutError and CallbackExternalError) and register_operation_error. # The upper bound guards one read of the SDK's internal ExecutionState, for # the error type that a MicroVM reports. See _reported_error_type. - "aws-durable-execution-sdk-python>=2.0.0,<4", + "aws-durable-execution-sdk-python>=3.0.0,<4", # botocore 1.43.35 is the first release with the lambda-microvms client. + # The bedrock-agentcore client has StopRuntimeSession since botocore + # 1.40.46, so this bound covers the agentcore operation too. # The core SDK requires only boto3 1.42.1. "boto3>=1.43.35", ] diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/__init__.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/__init__.py new file mode 100644 index 000000000..97ecb0ba2 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/__init__.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""Run durable jobs in Amazon Bedrock AgentCore Runtime sessions. + +This module is experimental. Its API can change in any release. + +The exports match ``@aws/durable-execution-sdk-js-extras/compute/agentcore``. +""" + +from aws_durable_execution_sdk_python_extras.compute.agentcore.config import ( + AgentcoreConfig, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.errors import ( + AgentcoreDeliveryError, + AgentcoreError, + AgentcoreJobFailedError, + AgentcoreTimeoutError, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.operation import ( + AgentcoreNotAcceptedError, + agentcore, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.retry import ( + default_agentcore_retry_strategy, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.subtypes import ( + AgentcoreOperationSubType, +) +from aws_durable_execution_sdk_python_extras.compute.errors import ComputeError +from aws_durable_execution_sdk_python_extras.compute.microvm.job import ( + MAX_JOB_INPUT_BYTES, +) + + +__all__ = [ + "MAX_JOB_INPUT_BYTES", + "AgentcoreConfig", + "AgentcoreDeliveryError", + "AgentcoreError", + "AgentcoreJobFailedError", + "AgentcoreNotAcceptedError", + "AgentcoreOperationSubType", + "AgentcoreTimeoutError", + "ComputeError", + "agentcore", + "default_agentcore_retry_strategy", +] diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/config.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/config.py new file mode 100644 index 000000000..0c215216c --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/config.py @@ -0,0 +1,117 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The configuration of :func:`agentcore`.""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.retries import RetryDecision + + +if TYPE_CHECKING: + from mypy_boto3_bedrock_agentcore import ( + BedrockAgentCoreClient as BedrockAgentcoreClient, + ) +else: + BedrockAgentcoreClient = Any + + +@dataclass(frozen=True, kw_only=True) +class AgentcoreConfig: + """The agent runtime of :func:`agentcore`, and how long to wait for its + result. + + :func:`agentcore` validates the config before it starts any durable + operation. + """ + + agent_runtime_arn: str + """The ARN of the AgentCore Runtime agent runtime that runs the job. + + The agent runtime must run the worker + ``aws_durable_execution_sdk_python_compute_worker.agentcore``, or code + that follows the same contract. Its execution role needs + ``lambda:SendDurableExecutionCallbackSuccess``, + ``lambda:SendDurableExecutionCallbackFailure``, and + ``lambda:SendDurableExecutionCallbackHeartbeat`` on the durable + function's ARN. + + The durable function's execution role needs + ``bedrock-agentcore:InvokeAgentRuntime`` and + ``bedrock-agentcore:StopRuntimeSession`` on two resources: this ARN, and + ``/runtime-endpoint/*``. A live run with only the runtime ARN + in the policy was denied on ``.../runtime-endpoint/DEFAULT``. To grant + one endpoint only, name the endpoint of ``qualifier`` instead of ``*``. + + The worker answers ``/ping`` with ``HealthyBusy`` while a job runs. A + live run kept a session through a 180-second job with an idle timeout + of 60 seconds. So the agent runtime's ``idleRuntimeSessionTimeout`` + does not end a running job. After the idle timeout, a session with no + running job ends. A later request with the same session ID then gets a + new microVM without an error, and the memory and files of the old one + are lost. This operation sends one job per session, so it does not + depend on state across that gap. + """ + + timeout: Duration + """The longest time to wait for the agent to report a result. + + It must be at least 1 second. AgentCore ends a session at the agent + runtime's ``maxLifetime``: at most 8 hours for a microVM session, and at + most 14 days on Instances. A job that runs longer than its session can + never report. Keep the timeout within the runtime's ``maxLifetime``. + """ + + qualifier: str | None = None + """The endpoint qualifier of the agent runtime, such as ``DEFAULT``. + + ``None`` sends no qualifier, and AgentCore uses its default endpoint. The + operation sends the same qualifier to ``InvokeAgentRuntime`` and + ``StopRuntimeSession``. + """ + + heartbeat_timeout: Duration | None = None + """The longest time between two heartbeats from the agent. + + Set it. It ends the job when the worker crashes or stops reporting. The + first period must cover the request step with its retries and a cold + session start, so 30 seconds or more is a safe value. See below. + + When it is set, an agent session that crashes fails the operation after + this time, instead of after ``timeout``. The agent receives the value in + the job document. + + The heartbeat timeout starts when the callback is created, before the + request step. The worker sends its first heartbeat when the job arrives. + So the first period must cover the request step with its retries and the + start of a new session. + """ + + retry_strategy: Callable[[Exception, int], RetryDecision] | None = None + """The step retry strategy of the request and stop steps. + + It is the second retry tier. The boto3 client already retries each call + inside one step attempt. Defaults to + :func:`default_agentcore_retry_strategy`. + + ``InvokeAgentRuntime`` has no idempotency token. So a retried request + step can deliver the same job twice. The worker answers a second + delivery of a callback ID that it already accepted without running the + job again. It remembers callback IDs only in its process memory. A new + process after the idle timeout runs a repeated job again. So the job + handler's side effects must be idempotent. + """ + + client: BedrockAgentcoreClient | None = None + """The boto3 ``bedrock-agentcore`` client. + + Defaults to one client per Region, created on first use. The Region comes + from the agent runtime ARN. The default client has a 5-second connect + timeout, a 60-second read timeout, and 3 attempts in botocore's standard + retry mode. A client that you pass keeps its own settings. + """ diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/errors.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/errors.py new file mode 100644 index 000000000..bdb6cea2d --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/errors.py @@ -0,0 +1,110 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The errors that the ``agentcore`` operation raises. + +Every AgentCore failure is an :class:`AgentcoreError`. Catch the base class to +handle every failure the same way, or a subclass to handle one kind. + +The operation raises the same class on the first run and on every replay. The +mechanism is the one of the MicroVM errors: + +1. Inside its child context, the operation raises a subclass, such as + :class:`AgentcoreDeliveryError`. +2. The SDK checkpoints the child context's failure with the class's + fully-qualified name as the error type. +3. The SDK then raises ``ChildContextError``, and rebuilds the escaping error + as ``__cause__`` from its registry of error types. +4. This module registers every AgentCore class with that registry. So + ``__cause__`` is the AgentCore class, and the operation raises it. + +The fully-qualified name is part of the checkpoint format. So the classes +must stay in this module, with these names. +""" + +from __future__ import annotations + +from aws_durable_execution_sdk_python.exceptions import ( + ChildContextError, + register_operation_error, +) + +from aws_durable_execution_sdk_python_extras.compute.errors import ComputeError + + +class AgentcoreError(ComputeError): + """The base class of every AgentCore failure. + + ``message`` names the operation and keeps the underlying message. + ``data`` and ``stack_trace`` are those of the failure, when it had any. + """ + + @classmethod + def from_child_context_error( + cls, name: str, error: ChildContextError + ) -> AgentcoreError: + """Return the AgentCore error that the operation's child context raised. + + 1. The SDK rebuilds a registered AgentCore error type as its own class, + in ``__cause__``. A copy of that error is returned. + 2. Any other error type becomes a plain :class:`AgentcoreError`. It + keeps the original error type in ``error_type``. + + The caller raises the result ``from error``. So the result must not be + ``error.__cause__`` itself. Otherwise each error is the other's + ``__cause__``, and a loop that follows ``__cause__`` never ends. + """ + cause = error.__cause__ + if isinstance(cause, AgentcoreError): + return type(cause)( + message=cause.message, + error_type=cause.error_type, + data=cause.data, + stack_trace=cause.stack_trace, + ) + return AgentcoreError( + message=f'AgentCore "{name}": {error.message}', + error_type=error.error_type, + data=error.data, + stack_trace=error.stack_trace, + ) + + +class AgentcoreDeliveryError(AgentcoreError): + """The job could not be delivered to the agent runtime. + + ``InvokeAgentRuntime`` failed after all retries, or the agent answered + something other than ``{"accepted": true}``. + """ + + +class AgentcoreJobFailedError(AgentcoreError): + """The job handler in the agent runtime failed. + + The agent reported the failure with + ``SendDurableExecutionCallbackFailure``. The message includes the error + type that the agent reported, such as the job handler's exception class. + + The operation also raises it when the agent sent a result that is not + valid JSON. + """ + + +class AgentcoreTimeoutError(AgentcoreError): + """No result arrived within the job's timeout, or no heartbeat arrived + within its heartbeat timeout. + + An agent session that crashed, or that AgentCore ended at its idle + timeout, ends this way. + """ + + +for _error_class in ( + AgentcoreError, + AgentcoreDeliveryError, + AgentcoreJobFailedError, + AgentcoreTimeoutError, +): + # Each class takes the four keywords that the registry passes: message, + # error_type, data, and stack_trace. + register_operation_error(_error_class) diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/job.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/job.py new file mode 100644 index 000000000..8fcba4f10 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/job.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The job document that the operation sends to the agent runtime. + +The document has the fields of the MicroVM job request, without +``microvmId``. So the worker parses both with the same code. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +from aws_durable_execution_sdk_python_extras.compute.microvm.job import ( + PAYLOAD_VERSION, + dumps, +) + + +@dataclass(frozen=True, kw_only=True) +class AgentcoreJobRequest: + """The ``payload`` of the ``InvokeAgentRuntime`` request. + + Wire names: ``version``, ``region``, ``callbackId``, + ``heartbeatTimeoutSeconds``, and ``input``. + """ + + region: str + """The Region of the durable function, for the callback API calls. + + The agent runtime can run in another Region than the durable function. + So the worker takes the Region of the callback calls from the document. + """ + callback_id: str + """The callback that the worker completes with the job's outcome.""" + job_input: Any + """The caller's input. It must be JSON-serializable.""" + heartbeat_timeout_seconds: int | None = None + """The heartbeat timeout, when the caller set one.""" + + def to_json(self) -> str: + """Serialize the request as compact JSON, as the MicroVM job request. + + Raises: + TypeError: When the input is not JSON-serializable. + ValueError: When the input holds ``nan`` or an infinite float. + """ + document: dict[str, Any] = { + "version": PAYLOAD_VERSION, + "region": self.region, + "callbackId": self.callback_id, + } + if self.heartbeat_timeout_seconds is not None: + document["heartbeatTimeoutSeconds"] = self.heartbeat_timeout_seconds + document["input"] = self.job_input + return dumps(document) diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/operation.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/operation.py new file mode 100644 index 000000000..dd0ea2ebd --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/operation.py @@ -0,0 +1,657 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The ``agentcore`` operation: run one job in a new AgentCore Runtime +session.""" + +from __future__ import annotations + +import json +import os +import threading +import time +import uuid +from collections.abc import Callable +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +import boto3 +from aws_durable_execution_sdk_python.config import ( + CallbackConfig, + ChildConfig, + Duration, + StepConfig, +) +from aws_durable_execution_sdk_python.exceptions import ( + CallbackError, + CallbackTimeoutError, + ChildContextError, + ExecutionError, + InvocationError, + StepError, + ValidationError, +) +from aws_durable_execution_sdk_python.retries import RetryDecision +from aws_durable_execution_sdk_python.serdes import JsonSerDes +from botocore.config import Config +from botocore.exceptions import ClientError + +from aws_durable_execution_sdk_python_extras.compute.agentcore.config import ( + AgentcoreConfig, + BedrockAgentcoreClient, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.errors import ( + AgentcoreDeliveryError, + AgentcoreError, + AgentcoreJobFailedError, + AgentcoreTimeoutError, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.job import ( + AgentcoreJobRequest, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.retry import ( + default_agentcore_retry_strategy, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore.subtypes import ( + AgentcoreOperationSubType, +) +from aws_durable_execution_sdk_python_extras.compute.microvm.job import ( + MAX_JOB_INPUT_BYTES, + dumps, +) +from aws_durable_execution_sdk_python_extras.compute.microvm.request import ( + INVOCATION_RESERVE_SECONDS, + call_before_deadline, + require_time_for_call, + reserve_starts_at, +) + + +if TYPE_CHECKING: + from aws_durable_execution_sdk_python.context import Callback, DurableContext + + +_CONNECT_TIMEOUT_SECONDS = 5 +_READ_TIMEOUT_SECONDS = 60 +_CLIENT_ATTEMPTS = 3 +"""The default client's limits. + +The first ``InvokeAgentRuntime`` of a session starts the session's microVM +before the agent answers. So the read timeout is 60 seconds, longer than the +30 seconds of the MicroVM client. One call can then take about 3 minutes. +The invocation reserve ends a step attempt earlier when the invocation has +less time left. See :func:`_call_before_reserve`. +""" + +MAX_ANSWER_BYTES = 64 * 1024 +"""The most bytes of the agent's answer that the request step reads. + +The worker answers ``{"accepted": true}``. The step reads no more than this +limit, so a long answer from other code does not fill the memory. +""" + + +class AgentcoreNotAcceptedError(Exception): + """The agent answered the job request with something other than + ``{"accepted": true}``. + + The default retry strategy does not retry it. The same agent gives the + same answer to the same job. + """ + + +def agentcore( + context: DurableContext, + name: str, + *, + job_input: Any, + config: AgentcoreConfig, +) -> Any: + """Run one job in a new AgentCore Runtime session, and return the job's + result. + + The operation runs its durable operations in one child context named + ``name``: + + 1. ``.session`` generates the runtime session ID, a UUID. The SDK + checkpoints it. So every retry and replay uses the same session. + 2. ``.callback`` creates a durable callback. + 3. ``.request`` calls ``InvokeAgentRuntime`` with the session ID + and the job document as the payload. AgentCore starts the session on + this first request. The agent answers ``{"accepted": true}`` at once, + and runs the job in the background. + 4. The child context waits on the callback. The invocation ends during + the wait, so no Lambda compute is billed. The agent completes the + callback with the job's result or failure. + 5. ``.stop`` calls ``StopRuntimeSession``. It runs after success, + failure, and timeout alike, and after a failed delivery. + + A stop failure does not fail the operation. The job's outcome is already + recorded when stop runs, and AgentCore ends an idle session at the agent + runtime's idle timeout. So the operation logs an error and returns the + job's outcome. + + The session lives only for this one job. After the idle timeout, the + same session ID would get a new microVM without its memory and files. + The operation never sends a second job to the session. So that loss + cannot affect a job. + + ``InvokeAgentRuntime`` has no idempotency token. A retried request step + can deliver the same job twice to the same session. The worker answers a + repeated callback ID with ``{"accepted": true}``, and does not run the + job again. The worker remembers callback IDs only in its process + memory. So this guarantee holds per worker process. A retry can arrive + after the idle timeout has replaced the session's process, for example + when an answer was lost after the job finished. The new process then + runs the same job again. So delivery is at-least-once across process + replacement, and the job handler's side effects must be idempotent. + + ``job_input`` and ``config`` are keyword-only, as in ``microvm``. + + Args: + context: The context to create the durable operations in. + name: The child context name. It also prefixes the inner operations. + job_input: The job's input. It must be JSON-serializable, and the + same on every replay. + config: The agent runtime and the timeouts. + + Returns: + The value that the agent passed to + ``SendDurableExecutionCallbackSuccess``, parsed as JSON. + + Raises: + ValidationError: When ``name`` or a timeout is not valid, or the + Region of the durable function cannot be determined. No durable + operation starts. + TypeError: When ``job_input`` is not JSON-serializable. No durable + operation starts. + ValueError: When the JSON of ``job_input`` is over + ``MAX_JOB_INPUT_BYTES`` UTF-8 bytes. No durable operation starts. + AgentcoreDeliveryError: When ``InvokeAgentRuntime`` fails after all + retries, or the agent does not accept the job. + AgentcoreJobFailedError: When the agent reports a failure, or sends a + result that is not valid JSON. + AgentcoreTimeoutError: When no result or heartbeat arrives in time. + """ + scope = _AgentcoreScope.create(context, name, job_input, config) + + def run(child: DurableContext) -> Any: + return scope.run_job(child, job_input) + + try: + return context.run_in_child_context( + run, + name=name, + config=ChildConfig(sub_type=AgentcoreOperationSubType.AGENTCORE), + ) + except ChildContextError as error: + raise AgentcoreError.from_child_context_error(name, error) from error + + +@dataclass(frozen=True, kw_only=True) +class _AgentcoreScope: + """The validated values of one :func:`agentcore` call.""" + + name: str + config: AgentcoreConfig + region: str + """The Region of the durable function.""" + timeout_seconds: int + heartbeat_timeout_seconds: int | None + retry_strategy: Callable[[Exception, int], RetryDecision] + + @classmethod + def create( + cls, + context: DurableContext, + name: str, + job_input: Any, + config: AgentcoreConfig, + ) -> _AgentcoreScope: + """Validate the call, before any durable operation starts. + + ``agent_runtime_arn`` and ``qualifier`` are not checked here. The + boto3 client and the service reject a value that is not valid. The + request step then fails, and the operation raises + :class:`AgentcoreDeliveryError`. + + Raises: + ValidationError: When ``name`` or a timeout is not valid, or the + Region cannot be determined. + TypeError: When ``job_input`` is not JSON-serializable. + ValueError: When the JSON of ``job_input`` is too large. + """ + if not isinstance(name, str) or not name: + msg = "an AgentCore operation requires a non-empty name" + raise ValidationError(msg) + timeout_seconds = config.timeout.to_seconds() + if timeout_seconds < 1: + msg = f'AgentCore "{name}": timeout must be at least 1 second' + raise ValidationError(msg) + heartbeat_timeout_seconds = ( + None + if config.heartbeat_timeout is None + else config.heartbeat_timeout.to_seconds() + ) + if heartbeat_timeout_seconds is not None and heartbeat_timeout_seconds < 1: + msg = f'AgentCore "{name}": heartbeat_timeout must be at least 1 second' + raise ValidationError(msg) + _validate_input(name, job_input) + region = _function_region(context) + if not region: + # The worker needs the Region for the callback calls. So the + # operation fails before any durable operation instead. + msg = ( + f'AgentCore "{name}": cannot determine the AWS Region. The ' + "durable execution ARN has none, and AWS_REGION is not set." + ) + raise ValidationError(msg) + return cls( + name=name, + config=config, + region=region, + timeout_seconds=timeout_seconds, + heartbeat_timeout_seconds=heartbeat_timeout_seconds, + retry_strategy=config.retry_strategy or default_agentcore_retry_strategy, + ) + + def run_job(self, child: DurableContext, job_input: Any) -> Any: + """Run the operation's durable operations in its child context.""" + name = self.name + session_id: str = child.step( + # A UUID has 36 characters. InvokeAgentRuntime requires a session + # ID of at least 33. + lambda _: str(uuid.uuid4()), + name=f"{name}.session", + # Plain JSON, so a reader of the execution history finds the + # session ID. + config=StepConfig( + serdes=JsonSerDes(), sub_type=AgentcoreOperationSubType.SESSION_ID + ), + ) + callback = child.create_callback( + name=f"{name}.callback", + config=CallbackConfig( + timeout=Duration.from_seconds(self.timeout_seconds), + heartbeat_timeout=Duration.from_seconds( + self.heartbeat_timeout_seconds or 0 + ), + # The worker sends the result as a JSON string. The default + # callback serdes would return that string unparsed. + serdes=JsonSerDes(), + sub_type=AgentcoreOperationSubType.CALLBACK, + ), + ) + try: + self._deliver(child, session_id, callback.callback_id, job_input) + except AgentcoreDeliveryError: + self._stop(child, session_id) + raise + + try: + result = callback.result() + except ExecutionError: + # The execution fails without a retry, and the core SDK records + # no outcome for the child context. A stop step recorded now + # would change that history. So the error passes through. + raise + except InvocationError as error: + # An SDK invocation error is not a job outcome. A retryable one + # makes Lambda run the invocation again, and the replay waits for + # the same session. So the session stays up for it. + if not error.is_retryable(): + self._stop(child, session_id) + raise + except Exception as error: + # SuspendExecution is a BaseException, so a wait for the callback + # passes through here without stopping the session. + self._stop(child, session_id) + raise _job_error(name, callback, error) from error + self._stop(child, session_id) + return result + + def _deliver( + self, + child: DurableContext, + session_id: str, + callback_id: str, + job_input: Any, + ) -> None: + """Run the request step, ``.request``. + + Raises: + AgentcoreDeliveryError: When the delivery fails after all retries, + or the agent does not accept the job. + """ + payload = ( + AgentcoreJobRequest( + region=self.region, + callback_id=callback_id, + job_input=job_input, + heartbeat_timeout_seconds=self.heartbeat_timeout_seconds, + ) + .to_json() + .encode() + ) + remaining_seconds = _remaining_seconds(child) + + def attempt(_: object) -> None: + request: dict[str, Any] = { + "agentRuntimeArn": self.config.agent_runtime_arn, + "runtimeSessionId": session_id, + "contentType": "application/json", + "accept": "application/json", + "payload": payload, + } + if self.config.qualifier is not None: + request["qualifier"] = self.config.qualifier + + def call() -> None: + # The answer is a stream. A stalled read must end before the + # reserve too. So the read runs inside the same call. + response = self._client().invoke_agent_runtime(**request) + _check_accepted(response) + + _call_before_reserve("InvokeAgentRuntime", call, remaining_seconds) + + try: + child.step( + attempt, + name=f"{self.name}.request", + config=StepConfig( + retry_strategy=self.retry_strategy, + sub_type=AgentcoreOperationSubType.REQUEST, + ), + ) + except StepError as error: + msg = ( + f'AgentCore "{self.name}": the job could not be delivered: ' + f"{error.message or error.error_type}" + ) + raise AgentcoreDeliveryError( + message=msg, data=error.data, stack_trace=error.stack_trace + ) from error + + def _stop(self, child: DurableContext, session_id: str) -> None: + """Run the stop step, and log a final failure instead of raising. + + The job's outcome is already recorded when stop runs. AgentCore ends + an idle session at the agent runtime's idle timeout. So a stop + failure must not discard the outcome. The step's failure is + checkpointed. So on replay the step fails again at once, with the + same outcome. + + Each attempt ends before the invocation reserve. A stalled call + then fails the attempt with a retryable ``TimeoutError``, instead of + running until Lambda ends the invocation. So the step records an + outcome, and a later replay returns the job's outcome. + """ + remaining_seconds = _remaining_seconds(child) + try: + child.step( + lambda _: _call_before_reserve( + "StopRuntimeSession", + lambda: self._stop_call(session_id), + remaining_seconds, + ), + name=f"{self.name}.stop", + config=StepConfig( + retry_strategy=self.retry_strategy, + sub_type=AgentcoreOperationSubType.STOP, + ), + ) + except StepError as error: + # The session keeps running, and is billed, until AgentCore ends + # it. A missing permission fails every stop this way. This log + # is the only sign of it. So it is an error, not a warning. The + # permission hint is added only for an AccessDenied answer. A + # stop that never ran, for example for lack of invocation time, + # names its own cause in the error. + detail = error.message or error.error_type + hint = ( + " A missing permission needs " + "bedrock-agentcore:StopRuntimeSession in the function's " + "execution role, on the runtime ARN and on " + "/runtime-endpoint/*." + if "AccessDenied" in (detail or "") + else "" + ) + child.logger.error( + f'AgentCore "{self.name}": StopRuntimeSession failed for ' + f"session {session_id}. The session keeps running until the " + "agent runtime's idle timeout or maximum lifetime ends it." + f"{hint} Error: {detail}", + extra={"runtimeSessionId": session_id, "error": error.message}, + ) + + def _stop_call(self, session_id: str) -> None: + request: dict[str, Any] = { + "agentRuntimeArn": self.config.agent_runtime_arn, + "runtimeSessionId": session_id, + } + if self.config.qualifier is not None: + request["qualifier"] = self.config.qualifier + try: + self._client().stop_runtime_session(**request) + except ClientError as error: + # The service answers 404 ResourceNotFoundException for a + # session that was stopped or never started, measured live. A + # session that the idle timeout ended answers 200. Either way, + # no session runs. + if error.response.get("Error", {}).get("Code") != ( + "ResourceNotFoundException" + ): + raise + + def _client(self) -> BedrockAgentcoreClient: + if self.config.client is not None: + return self.config.client + return _DEFAULT_CLIENTS.get(_arn_region(self.config.agent_runtime_arn)) + + +def _check_accepted(response: Any) -> None: + """Read the agent's answer, and check that it accepted the job. + + The answer of ``InvokeAgentRuntime`` is a stream in ``response``. The + step reads at most :data:`MAX_ANSWER_BYTES` of it, and closes it. An + unread stream would hold its connection. + + Raises: + AgentcoreNotAcceptedError: When the answer is not + ``{"accepted": true}``. + """ + stream = response["response"] + try: + body = stream.read(MAX_ANSWER_BYTES) + finally: + stream.close() + try: + answer = json.loads(body) + except ValueError: + answer = None + if not isinstance(answer, dict) or answer.get("accepted") is not True: + text = body[:200].decode(errors="replace") + msg = f'the agent did not answer {{"accepted": true}}. It answered: {text}' + raise AgentcoreNotAcceptedError(msg) + + +def _job_error(name: str, callback: Callback, error: Exception) -> Exception: + """Convert a failure of the callback wait into its AgentCore error.""" + if isinstance(error, CallbackTimeoutError): + return AgentcoreTimeoutError( + message=f'AgentCore "{name}": the job timed out: {error.message}', + data=error.data, + stack_trace=error.stack_trace, + ) + if isinstance(error, CallbackError): + reported = _reported_error_type(callback) + detail = f" ({reported})" if reported else "" + return AgentcoreJobFailedError( + message=f'AgentCore "{name}": the job failed{detail}: {error.message}', + data=error.data, + stack_trace=error.stack_trace, + ) + # Any other failure, such as a result that is not valid JSON. The agent + # sent the result, so the job failed. + detail = getattr(error, "message", None) or str(error) + return AgentcoreJobFailedError( + message=( + f'AgentCore "{name}": the job result could not be read ' + f"({type(error).__name__}): {detail}" + ) + ) + + +def _reported_error_type(callback: Callback) -> str | None: + """The error type that the agent reported for a failed callback. + + ``CallbackExternalError`` carries the message and the data, not the type. + The callback's checkpoint holds the type. ``ExecutionState`` is internal + to the SDK. The MicroVM operation reads it the same way, and the + package's dependency has an upper bound on the SDK's major version. + + ``Error`` is the type that a worker reports for a value that is not an + exception. It names nothing, so it is left out. + """ + try: + error = callback.state.get_checkpoint_result(callback.operation_id).error + reported = None if error is None else error.type + except (AttributeError, KeyError, TypeError): + return None + if not isinstance(reported, str) or reported in {"", "Error"}: + return None + return reported + + +def _validate_input(name: str, job_input: Any) -> None: + """Check that the job input is JSON-serializable, and fits in a job body. + + The worker accepts a job body of at most 6 MiB, as the MicroVM worker + does. So the limit is the MicroVM limit, :data:`MAX_JOB_INPUT_BYTES`. + ``InvokeAgentRuntime`` accepts a payload of up to 100 MB, so the worker + limit is the smaller one. + + Raises: + TypeError: When it is not JSON-serializable. + ValueError: When its JSON is over :data:`MAX_JOB_INPUT_BYTES` bytes. + """ + try: + text = dumps(job_input) + except (TypeError, ValueError) as error: + msg = f'AgentCore "{name}": the job input must be JSON-serializable: {error}' + raise TypeError(msg) from error + size = len(text.encode()) + if size > MAX_JOB_INPUT_BYTES: + msg = ( + f'AgentCore "{name}": the job input is {size} bytes as JSON. The ' + f"limit is {MAX_JOB_INPUT_BYTES} bytes, because the worker accepts a " + "job body of at most 6 MiB. Pass larger data by reference, for " + "example as an S3 key." + ) + raise ValueError(msg) + + +def _remaining_seconds(context: DurableContext) -> Callable[[], float | None]: + """Return a reader of the invocation's remaining time, in seconds. + + ``None`` means that the compute reports no deadline. The MicroVM + operation reads the time the same way. + """ + lambda_context = context.lambda_context + if lambda_context is None: + return lambda: None + return lambda: lambda_context.get_remaining_time_in_millis() / 1000 + + +def _call_before_reserve( + api: str, + call: Callable[[], None], + remaining_seconds: Callable[[], float | None], +) -> None: + """Run one call of a step attempt, and end it before the invocation + reserve. + + 1. A Lambda timeout during a step attempt records no outcome for the + attempt. So replay runs the attempt again, and a call that always + stalls would never record an outcome. + 2. The MicroVM operation keeps an invocation reserve of + :data:`INVOCATION_RESERVE_SECONDS` for that reason. This function + uses the same reserve and the same helper, ``call_before_deadline``. + 3. A call needs at least 11 seconds left in the invocation: the 10 + seconds of the reserve, and 1 second for the call. With less, the + MicroVM preflight ``require_time_for_call`` fails the attempt before + any API call, and its message asks for a longer function timeout. + 4. A call that has not ended at the reserve fails the attempt with + ``TimeoutError``. The default retry strategy retries it, in a later + invocation. boto3 cannot cancel a call, so the call's thread keeps + running, and its result is discarded. + + Raises: + MicrovmEndpointUnavailableError: When less than 11 seconds remain. + TimeoutError: When the call does not end before the reserve. + Exception: The error of the call. + """ + require_time_for_call(remaining_seconds) + started = time.monotonic() + + def expired() -> Exception: + elapsed_ms = round((time.monotonic() - started) * 1000) + return TimeoutError( + f"{api} did not end before the invocation reserve, after " + f"{elapsed_ms} ms. The step stops " + f"{INVOCATION_RESERVE_SECONDS:.0f} seconds before the Lambda " + "invocation times out, so that it can record its outcome." + ) + + call_before_deadline(call, reserve_starts_at(remaining_seconds), expired) + + +def _function_region(context: DurableContext) -> str: + """The Region of the durable execution, or ``AWS_REGION``.""" + return _arn_region(context.execution_context.durable_execution_arn) or ( + os.environ.get("AWS_REGION", "") + ) + + +def _arn_region(arn: str) -> str: + """The Region of an ARN of the form ``arn::::...``, + or ``""``.""" + parts = arn.split(":") + if len(parts) > 3 and parts[0] == "arn": + return parts[3] + return "" + + +class _DefaultClients: + """One ``bedrock-agentcore`` client per Region, created on first use. + + A boto3 client is safe to use from several threads. Creating one from a + shared session is not. So creation runs under a lock, and each client + gets its own session. The key ``""`` gets boto3's default Region. + """ + + def __init__(self) -> None: + self._lock = threading.Lock() + self._clients: dict[str, BedrockAgentcoreClient] = {} + + def get(self, region: str) -> BedrockAgentcoreClient: + with self._lock: + client = self._clients.get(region) + if client is None: + client = boto3.session.Session().client( + "bedrock-agentcore", + region_name=region or None, + config=Config( + connect_timeout=_CONNECT_TIMEOUT_SECONDS, + read_timeout=_READ_TIMEOUT_SECONDS, + retries={ + "mode": "standard", + "total_max_attempts": _CLIENT_ATTEMPTS, + }, + ), + ) + self._clients[region] = client + return client + + +_DEFAULT_CLIENTS = _DefaultClients() diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/retry.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/retry.py new file mode 100644 index 000000000..070d94d16 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/retry.py @@ -0,0 +1,91 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The default step retry strategy of the ``agentcore`` operation.""" + +from __future__ import annotations + +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.retries import RetryDecision +from botocore.exceptions import ClientError, HTTPClientError +from botocore.exceptions import ConnectionError as BotocoreConnectionError + +from aws_durable_execution_sdk_python_extras.compute.microvm.request import ( + MicrovmEndpointUnavailableError, +) + + +MAX_ATTEMPTS = 5 +"""Attempts of one step, the first one included.""" + +MAX_DELAY_SECONDS = 60 +"""The longest delay between two attempts.""" + +_RETRYABLE_ERROR_CODES = frozenset( + { + # The error codes of the bedrock-agentcore model that a later + # attempt can fix. + "ThrottlingException", + "InternalServerException", + "RetryableConflictException", + # The transient codes of botocore's standard retry mode. + "RequestTimeout", + "RequestTimeoutException", + "PriorRequestNotComplete", + } +) + +_RETRYABLE_ERROR_TYPES: tuple[type[Exception], ...] = ( + TimeoutError, + ConnectionError, + # botocore's connection and HTTP errors do not subclass the builtin + # ConnectionError. botocore's standard retry mode treats these two bases + # as transient. + BotocoreConnectionError, + HTTPClientError, + # The preflight before each call raises it when the invocation has less + # than 11 seconds left. A retry runs early in a new invocation, as in + # the MicroVM operation. + MicrovmEndpointUnavailableError, +) + + +def default_agentcore_retry_strategy(error: Exception, attempt: int) -> RetryDecision: + """Retry a step that failed for a reason that a later attempt can fix. + + The request and stop steps use this strategy unless the config sets + ``retry_strategy``. Inside each attempt, the boto3 client already retried + throttling and server errors. + + An error is retryable when it is one of these: + + 1. A service error whose code shows a transient failure, such as + ``ThrottlingException`` or ``RetryableConflictException``. + 2. A service error with HTTP status 429, or 500 and above. + 3. A connection failure or a timeout. + + Every other error fails the step at once. A ``ValidationException`` or + an ``AccessDeniedException``, for example, fails the same way on every + attempt. + + Returns: + A retry after 2 to the power ``attempt`` seconds, at most 60 seconds, + for at most 5 attempts in all. Otherwise no retry. + """ + if attempt >= MAX_ATTEMPTS or not _is_retryable(error): + return RetryDecision.no_retry() + return RetryDecision.retry( + Duration.from_seconds(min(2**attempt, MAX_DELAY_SECONDS)) + ) + + +def _is_retryable(error: Exception) -> bool: + if isinstance(error, _RETRYABLE_ERROR_TYPES): + return True + if not isinstance(error, ClientError): + return False + response = error.response + if response.get("Error", {}).get("Code") in _RETRYABLE_ERROR_CODES: + return True + status = response.get("ResponseMetadata", {}).get("HTTPStatusCode") + return isinstance(status, int) and (status == 429 or status >= 500) diff --git a/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/subtypes.py b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/subtypes.py new file mode 100644 index 000000000..3387d0566 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/compute/agentcore/subtypes.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The subtypes that :func:`agentcore` records on its durable operations.""" + +from __future__ import annotations + +from enum import StrEnum + + +class AgentcoreOperationSubType(StrEnum): + """The subtypes that ``agentcore`` records on its durable operations. + + Execution history and plugins can filter on them without parsing + operation names. The values equal ``AgentcoreOperationSubType`` in the + JavaScript SDK. + + Replay compares each subtype with the checkpoint. So a changed value + breaks the executions in flight, and these values must stay fixed. + """ + + AGENTCORE = "Agentcore" + """The child context of an ``agentcore`` call.""" + SESSION_ID = "AgentcoreSessionId" + """The step that generates and checkpoints the runtime session ID.""" + CALLBACK = "AgentcoreCallback" + """The callback that the agent completes with the job result.""" + REQUEST = "AgentcoreRequest" + """The step that calls ``InvokeAgentRuntime`` with the job.""" + STOP = "AgentcoreStop" + """The step that calls ``StopRuntimeSession``.""" diff --git a/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_errors_test.py b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_errors_test.py new file mode 100644 index 000000000..69cd9ed26 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_errors_test.py @@ -0,0 +1,66 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""Tests for rebuilding AgentCore errors from a child context failure.""" + +from __future__ import annotations + +import pytest +from aws_durable_execution_sdk_python.exceptions import ChildContextError +from aws_durable_execution_sdk_python.lambda_service import ErrorObject + +from aws_durable_execution_sdk_python_extras.compute.agentcore import ( + AgentcoreDeliveryError, + AgentcoreError, + AgentcoreJobFailedError, + AgentcoreTimeoutError, + ComputeError, +) + + +def child_context_error(raised: Exception) -> ChildContextError: + """The error that the SDK raises for a child context that raised + ``raised``, built from the checkpoint fields as the SDK builds it.""" + try: + ErrorObject.from_exception(raised).raise_as_operation_error(ChildContextError) + except ChildContextError as error: + return error + msg = "unreachable" + raise AssertionError(msg) + + +@pytest.mark.parametrize( + "error_class", + [ + AgentcoreError, + AgentcoreDeliveryError, + AgentcoreJobFailedError, + AgentcoreTimeoutError, + ], +) +def test_rebuilds_each_agentcore_error_class( + error_class: type[AgentcoreError], +) -> None: + raised = error_class( + message='AgentCore "review": broken', data='{"k":1}', stack_trace=["at x"] + ) + + rebuilt = AgentcoreError.from_child_context_error( + "review", child_context_error(raised) + ) + + assert type(rebuilt) is error_class + assert isinstance(rebuilt, ComputeError) + assert rebuilt.message == 'AgentCore "review": broken' + assert rebuilt.data == '{"k":1}' + assert rebuilt is not child_context_error(raised).__cause__ + + +def test_wraps_another_error_type_in_the_base_class() -> None: + rebuilt = AgentcoreError.from_child_context_error( + "review", child_context_error(RuntimeError("boom")) + ) + + assert type(rebuilt) is AgentcoreError + assert rebuilt.message == 'AgentCore "review": boom' + assert rebuilt.error_type == "RuntimeError" diff --git a/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_operation_test.py b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_operation_test.py new file mode 100644 index 000000000..1d956bb37 --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_operation_test.py @@ -0,0 +1,853 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""Tests for the agentcore operation, run on the local test runner.""" + +from __future__ import annotations + +import io +import json +import threading +import time +from collections.abc import Callable, Iterator +from types import SimpleNamespace +from typing import Any + +import boto3 +import pytest +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import Callback, DurableContext +from aws_durable_execution_sdk_python.exceptions import ( + InvocationError, + NonDeterministicExecutionError, + ValidationError, +) +from aws_durable_execution_sdk_python.execution import ( + InvocationStatus, + durable_execution, +) +from aws_durable_execution_sdk_python.lambda_service import ErrorObject +from aws_durable_execution_sdk_python.retries import RetryDecision +from aws_durable_execution_sdk_python_testing.runner import ( + DurableFunctionTestResult, + DurableFunctionTestRunner, +) +from botocore.exceptions import ClientError +from botocore.response import StreamingBody +from botocore.stub import ANY, Stubber + +from aws_durable_execution_sdk_python_extras.compute.agentcore import ( + MAX_JOB_INPUT_BYTES, + AgentcoreConfig, + AgentcoreDeliveryError, + AgentcoreError, + AgentcoreNotAcceptedError, + AgentcoreOperationSubType, + ComputeError, + agentcore, + default_agentcore_retry_strategy, +) +from aws_durable_execution_sdk_python_extras.compute.agentcore import operation +from aws_durable_execution_sdk_python_extras.compute.microvm.request import ( + INVOCATION_RESERVE_SECONDS, + MicrovmEndpointUnavailableError, +) + + +WAIT = 30 +ARN = "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/reviewer-abc123" + + +def agentcore_error(code: str, status: int = 400) -> ClientError: + response: Any = { + "Error": {"Code": code, "Message": f"{code} message"}, + "ResponseMetadata": {"HTTPStatusCode": status}, + } + return ClientError(response, "InvokeAgentRuntime") + + +def config_with(client: Any, **fields: Any) -> AgentcoreConfig: + values: dict[str, Any] = { + "agent_runtime_arn": ARN, + "timeout": Duration.from_minutes(5), + "client": client, + # One quick retry keeps the retry tests fast. + "retry_strategy": lambda error, attempt: ( + RetryDecision.retry(Duration.from_seconds(1)) + if attempt < 2 + and getattr(error, "response", {}).get("Error", {}).get("Code") + == "ThrottlingException" + else RetryDecision.no_retry() + ), + **fields, + } + return AgentcoreConfig(**values) + + +def outcome(result: DurableFunctionTestResult) -> Any: + assert result.status is InvocationStatus.SUCCEEDED, result.error + return json.loads(result.result) if result.result is not None else None + + +Complete = Callable[[DurableFunctionTestRunner, str], None] + + +@pytest.fixture +def run() -> Iterator[Callable[..., Any]]: + """Run a handler on the local test runner, and close the runner after.""" + runners: list[DurableFunctionTestRunner] = [] + + def _run( + handler: Callable[..., Any], event: Any, complete: Complete | None = None + ) -> DurableFunctionTestResult: + runner = DurableFunctionTestRunner(handler=handler, poll_interval=0.05) + runners.append(runner) + arn = runner.run_async(input=json.dumps(event), execution_timeout=WAIT) + if complete is not None: + callback_id = runner.wait_for_callback( + arn, name="review.callback", timeout=WAIT + ) + complete(runner, callback_id) + return runner.wait_for_result(arn, timeout=WAIT) + + yield _run + for runner in runners: + runner.close() + + +def succeed_with(value: Any) -> Complete: + def complete(runner: DurableFunctionTestRunner, callback_id: str) -> None: + runner.send_callback_success(callback_id, json.dumps(value).encode()) + + return complete + + +def fail_with(error_type: str | None, message: str) -> Complete: + def complete(runner: DurableFunctionTestRunner, callback_id: str) -> None: + runner.send_callback_failure( + callback_id, + ErrorObject(message=message, type=error_type, data=None, stack_trace=None), + ) + + return complete + + +def catching(config: AgentcoreConfig) -> Callable[..., Any]: + """A handler that returns the class and message of an AgentCore error.""" + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + return { + "result": agentcore(context, "review", job_input=event, config=config) + } + except AgentcoreError as error: + return {"error": type(error).__name__, "message": error.message} + + return handler + + +def test_delivers_the_job_returns_the_result_and_stops_the_session( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + config = config_with(client, heartbeat_timeout=Duration.from_seconds(60)) + + result = run(catching(config), {"pr": 7}, succeed_with({"approved": True})) + + assert outcome(result) == {"result": {"approved": True}} + [invoke] = client.requests("invoke_agent_runtime") + session_id = invoke["runtimeSessionId"] + assert len(session_id) >= 33 + payload = json.loads(invoke.pop("payload")) + assert invoke == { + "agentRuntimeArn": ARN, + "runtimeSessionId": session_id, + "contentType": "application/json", + "accept": "application/json", + } + assert payload == { + "version": 1, + "region": "us-west-2", + "callbackId": payload["callbackId"], + "heartbeatTimeoutSeconds": 60, + "input": {"pr": 7}, + } + assert client.requests("stop_runtime_session") == [ + {"agentRuntimeArn": ARN, "runtimeSessionId": session_id} + ] + assert [stream.closed for stream in client.streams] == [True] + + +def test_records_the_operations_in_one_child_context( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + + result = run(catching(config_with(client)), {}, succeed_with(1)) + + context = result.get_context("review") + assert [child.name for child in context.child_operations] == [ + "review.session", + "review.callback", + "review.request", + "review.stop", + ] + session = context.get_step("review.session") + assert session.result is not None + [invoke] = client.requests("invoke_agent_runtime") + assert json.loads(session.result) == invoke["runtimeSessionId"] + + +def test_passes_the_qualifier_to_both_calls( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + + run(catching(config_with(client, qualifier="DEFAULT")), {}, succeed_with(1)) + + [invoke] = client.requests("invoke_agent_runtime") + [stop] = client.requests("stop_runtime_session") + assert invoke["qualifier"] == "DEFAULT" + assert stop["qualifier"] == "DEFAULT" + + +def test_replay_and_a_retried_request_reuse_the_session_id( + run: Any, fake_agentcore_client: Any +) -> None: + # The callback wait ends the first invocation. The stop step runs in a + # replay, so its session ID comes from the session step's checkpoint. + client = fake_agentcore_client( + invokes=[agentcore_error("ThrottlingException", 429)] + ) + + result = run(catching(config_with(client)), {}, succeed_with("ok")) + + assert outcome(result) == {"result": "ok"} + invokes = client.requests("invoke_agent_runtime") + [stop] = client.requests("stop_runtime_session") + assert len(invokes) == 2 + assert {invoke["runtimeSessionId"] for invoke in invokes} == { + stop["runtimeSessionId"] + } + assert invokes[0]["payload"] == invokes[1]["payload"] + + +def test_fails_with_the_job_failed_error_and_still_stops( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + + result = run( + catching(config_with(client)), {}, fail_with("RuntimeError", "3 tests failed") + ) + + assert outcome(result) == { + "error": "AgentcoreJobFailedError", + "message": 'AgentCore "review": the job failed (RuntimeError): 3 tests failed', + } + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_fails_with_the_job_failed_error_for_a_result_that_is_not_json( + run: Any, fake_agentcore_client: Any +) -> None: + def complete(runner: DurableFunctionTestRunner, callback_id: str) -> None: + runner.send_callback_success(callback_id, b"not json") + + result = run(catching(config_with(fake_agentcore_client())), {}, complete) + + assert outcome(result)["error"] == "AgentcoreJobFailedError" + + +def test_fails_with_a_timeout_error_when_no_result_arrives( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + config = config_with(client, timeout=Duration.from_seconds(2)) + + result = run(catching(config), {}) + + error = outcome(result) + assert error["error"] == "AgentcoreTimeoutError" + assert error["message"].startswith('AgentCore "review": the job timed out: ') + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_fails_with_a_delivery_error_and_stops_when_the_request_fails( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client(invokes=[agentcore_error("AccessDeniedException")]) + + result = run(catching(config_with(client)), {}) + + error = outcome(result) + assert error["error"] == "AgentcoreDeliveryError" + assert error["message"].startswith( + 'AgentCore "review": the job could not be delivered: ' + ) + assert "AccessDeniedException" in error["message"] + assert len(client.requests("invoke_agent_runtime")) == 1 + assert len(client.requests("stop_runtime_session")) == 1 + + +@pytest.mark.parametrize("answer", [b'{"accepted":false}', b"[]", b"not json", b""]) +def test_fails_with_a_delivery_error_when_the_agent_does_not_accept( + run: Any, fake_agentcore_client: Any, answer: bytes +) -> None: + client = fake_agentcore_client(invokes=[answer]) + + result = run(catching(config_with(client)), {}) + + error = outcome(result) + assert error["error"] == "AgentcoreDeliveryError" + assert 'did not answer {"accepted": true}' in error["message"] + # The default strategy does not retry it, and neither does the test's. + assert len(client.requests("invoke_agent_runtime")) == 1 + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_returns_the_result_when_stop_fails( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client(stops=[agentcore_error("AccessDeniedException")]) + + result = run(catching(config_with(client)), {}, succeed_with("done")) + + assert outcome(result) == {"result": "done"} + stop = result.get_context("review").get_step("review.stop") + assert stop.error is not None + + +def test_keeps_the_job_error_when_stop_fails( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client(stops=[agentcore_error("AccessDeniedException")]) + + result = run(catching(config_with(client)), {}, fail_with("ValueError", "bad")) + + assert outcome(result)["error"] == "AgentcoreJobFailedError" + + +def test_treats_resource_not_found_from_stop_as_stopped( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client( + stops=[agentcore_error("ResourceNotFoundException", 404)] + ) + + result = run(catching(config_with(client)), {}, succeed_with("done")) + + assert outcome(result) == {"result": "done"} + assert result.get_context("review").get_step("review.stop").error is None + + +def test_raises_the_same_class_after_the_handler_stops_catching( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client(invokes=[agentcore_error("ValidationException")]) + config = config_with(client) + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + agentcore(context, "review", job_input=event, config=config) + except ComputeError as error: + return type(error).__name__ + return None + + assert outcome(run(handler, {})) == "AgentcoreDeliveryError" + + +@pytest.mark.parametrize( + ("name", "fields", "match"), + [ + ("", {}, "non-empty name"), + ("review", {"timeout": Duration.from_seconds(0)}, "timeout must be at least"), + ( + "review", + {"heartbeat_timeout": Duration.from_seconds(0)}, + "heartbeat_timeout must be at least", + ), + ], +) +def test_rejects_a_config_that_is_not_valid_before_any_operation( + fake_agentcore_client: Any, name: str, fields: dict[str, Any], match: str +) -> None: + client = fake_agentcore_client() + context: Any = None + + with pytest.raises(ValidationError, match=match): + agentcore(context, name, job_input={}, config=config_with(client, **fields)) + assert client.calls == [] + + +def test_rejects_a_job_input_over_the_size_limit_before_the_request( + run: Any, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + config = config_with(client) + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + agentcore( + context, + "review", + job_input="x" * (MAX_JOB_INPUT_BYTES - 1), + config=config, + ) + except ValueError as error: + return str(error) + return None + + message = outcome(run(handler, {})) + assert "the job input is" in message + assert client.calls == [] + + +def test_rejects_a_job_input_that_is_not_json( + run: Any, fake_agentcore_client: Any +) -> None: + config = config_with(fake_agentcore_client()) + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + agentcore(context, "review", job_input={1, 2}, config=config) + except TypeError as error: + return str(error) + return None + + assert "JSON-serializable" in outcome(run(handler, {})) + + +def test_requests_match_the_bedrock_agentcore_api_model(run: Any) -> None: + # A real boto3 client with a Stubber validates each request against the + # botocore model, and the answer has a real streaming body. + client = boto3.client( + "bedrock-agentcore", + region_name="us-west-2", + aws_access_key_id="test", + aws_secret_access_key="test", + ) + stubber = Stubber(client) + answer = b'{"accepted": true}' + stubber.add_response( + "invoke_agent_runtime", + { + "response": StreamingBody(io.BytesIO(answer), len(answer)), + "contentType": "application/json", + "statusCode": 200, + }, + { + "agentRuntimeArn": ARN, + "runtimeSessionId": ANY, + "qualifier": "DEFAULT", + "contentType": "application/json", + "accept": "application/json", + "payload": ANY, + }, + ) + stubber.add_response( + "stop_runtime_session", + {"statusCode": 200}, + {"agentRuntimeArn": ARN, "runtimeSessionId": ANY, "qualifier": "DEFAULT"}, + ) + with stubber: + result = run( + catching(config_with(client, qualifier="DEFAULT")), + {}, + succeed_with("ok"), + ) + stubber.assert_no_pending_responses() + + assert outcome(result) == {"result": "ok"} + + +def test_uses_the_default_client_of_the_runtime_region( + run: Any, monkeypatch: pytest.MonkeyPatch, fake_agentcore_client: Any +) -> None: + client = fake_agentcore_client() + regions: list[str] = [] + + class Clients: + def get(self, region: str) -> Any: + regions.append(region) + return client + + monkeypatch.setattr(operation, "_DEFAULT_CLIENTS", Clients()) + arn = ARN.replace("us-west-2", "eu-west-1") + + result = run( + catching(config_with(None, agent_runtime_arn=arn)), {}, succeed_with(1) + ) + + assert outcome(result) == {"result": 1} + assert set(regions) == {"eu-west-1"} + # The callback calls go to the durable function's Region. + [invoke] = client.requests("invoke_agent_runtime") + assert json.loads(invoke["payload"])["region"] == "us-west-2" + + +def test_default_clients_creates_one_client_per_region() -> None: + clients = operation._DefaultClients() # noqa: SLF001 + + first = clients.get("us-west-2") + assert clients.get("us-west-2") is first + assert first.meta.region_name == "us-west-2" + assert clients.get("eu-west-1") is not first + + +@pytest.mark.parametrize( + ("error", "retried"), + [ + (agentcore_error("ThrottlingException", 429), True), + (agentcore_error("RetryableConflictException", 409), True), + (agentcore_error("InternalServerException", 500), True), + (agentcore_error("SomethingElse", 503), True), + (agentcore_error("ValidationException", 400), False), + (agentcore_error("AccessDeniedException", 403), False), + (agentcore_error("RuntimeClientError", 424), False), + (TimeoutError(), True), + (AgentcoreNotAcceptedError("no"), False), + (AgentcoreDeliveryError("no"), False), + ], +) +def test_default_retry_strategy(error: Exception, retried: bool) -> None: + assert default_agentcore_retry_strategy(error, 1).should_retry is retried + assert not default_agentcore_retry_strategy(error, 5).should_retry + + +@pytest.mark.parametrize("error_type", [None, "Error"]) +def test_leaves_out_an_error_type_that_names_nothing( + run: Any, fake_agentcore_client: Any, error_type: str | None +) -> None: + result = run( + catching(config_with(fake_agentcore_client())), + {}, + fail_with(error_type, "broken"), + ) + + assert outcome(result)["message"] == 'AgentCore "review": the job failed: broken' + + +def test_leaves_out_the_error_type_when_the_checkpoint_shape_changed() -> None: + # The checkpoint state is internal to the SDK, and can change in a + # release. + callback: Any = SimpleNamespace(state=object(), operation_id="op") + + assert operation._reported_error_type(callback) is None # noqa: SLF001 + + +def test_rejects_a_call_without_a_region( + run: Any, fake_agentcore_client: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + # The local runner's execution ARN is not an ARN. So without AWS_REGION + # the operation has no Region for the job document. + monkeypatch.delenv("AWS_REGION") + client = fake_agentcore_client() + config = config_with(client) + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + agentcore(context, "review", job_input={}, config=config) + except ValidationError as error: + return str(error) + return None + + assert "cannot determine the AWS Region" in outcome(run(handler, {})) + assert client.calls == [] + + +def test_default_clients_uses_the_default_region_for_an_arn_without_one( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("AWS_DEFAULT_REGION", "ap-south-1") + clients = operation._DefaultClients() # noqa: SLF001 + + assert clients.get("").meta.region_name == "ap-south-1" + + +@pytest.mark.parametrize("retryable", [True, False]) +def test_passes_an_invocation_error_of_the_wait_through( + fake_agentcore_client: Any, monkeypatch: pytest.MonkeyPatch, retryable: bool +) -> None: + # An SDK invocation error is not a job outcome. A retryable one makes + # Lambda run the invocation again, and the replay waits for the same + # session. So only an error that is not retryable stops it. + class Interrupted(InvocationError): + def is_retryable(self) -> bool: + return retryable + + original = Callback.result + raised = threading.Event() + + def result_once(self: Any) -> Any: + if not raised.is_set(): + raised.set() + msg = "interrupted" + raise Interrupted(msg) + return original(self) + + monkeypatch.setattr(Callback, "result", result_once) + order: list[str] = [] + client = fake_agentcore_client(stops=[lambda request: order.append("stop")]) + runner = DurableFunctionTestRunner( + handler=catching(config_with(client)), poll_interval=0.05 + ) + try: + arn = runner.run_async(input="{}", execution_timeout=WAIT) + if retryable: + callback_id = runner.wait_for_callback( + arn, name="review.callback", timeout=WAIT + ) + # A stop for the error would run at once after the raise, in the + # same thread. So 0.5 seconds after it, the order shows whether + # one ran. + assert raised.wait(WAIT) + time.sleep(0.5) + order.append("complete") + runner.send_callback_success(callback_id, b'"done"') + result = runner.wait_for_result(arn, timeout=WAIT) + finally: + runner.close() + + assert outcome(result) == ( + {"result": "done"} + if retryable + else {"error": "AgentcoreError", "message": 'AgentCore "review": interrupted'} + ) + assert order == (["complete", "stop"] if retryable else ["stop"]) + + +def test_passes_an_execution_error_of_the_wait_through( + fake_agentcore_client: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + # The core SDK records no outcome for an ExecutionError, and the + # execution fails without a retry. A stop step recorded for it would + # change that history. + def result_fails(self: Any) -> Any: + msg = "history does not match" + raise NonDeterministicExecutionError(msg) + + monkeypatch.setattr(Callback, "result", result_fails) + client = fake_agentcore_client() + runner = DurableFunctionTestRunner( + handler=catching(config_with(client)), poll_interval=0.05 + ) + try: + arn = runner.run_async(input="{}", execution_timeout=WAIT) + result = runner.wait_for_result(arn, timeout=WAIT) + finally: + runner.close() + + assert result.status is InvocationStatus.FAILED + assert result.error is not None + assert "history does not match" in (result.error.message or "") + assert client.requests("stop_runtime_session") == [] + + +def test_reads_only_the_first_64_kib_of_the_answer( + run: Any, fake_agentcore_client: Any +) -> None: + # The accepted document starts after 64 KiB of spaces. A read without + # the limit would parse it and accept the job. + answer = b" " * (64 * 1024) + b'{"accepted":true}' + client = fake_agentcore_client(invokes=[answer]) + + result = run(catching(config_with(client)), {}) + + assert outcome(result)["error"] == "AgentcoreDeliveryError" + [stream] = client.streams + assert stream.closed + assert stream.read() == b'{"accepted":true}' + + +RESERVE_IN = 1.5 +"""The seconds from the start of a call to the invocation reserve, in the +reserve tests.""" + + +@pytest.fixture +def near_reserve(monkeypatch: pytest.MonkeyPatch) -> None: + """Make every invocation report that the reserve starts in + :data:`RESERVE_IN` seconds.""" + monkeypatch.setattr( + operation, + "_remaining_seconds", + lambda context: lambda: INVOCATION_RESERVE_SECONDS + RESERVE_IN, + ) + + +@pytest.fixture +def stall() -> Iterator[Callable[[dict[str, Any]], None]]: + """An answer that blocks the call until the test ends.""" + release = threading.Event() + + def _stall(request: dict[str, Any]) -> None: + release.wait(WAIT) + + yield _stall + release.set() + + +def test_ends_a_stalled_request_before_the_reserve_and_retries_it( + run: Any, fake_agentcore_client: Any, near_reserve: None, stall: Any +) -> None: + # The default strategy retries the TimeoutError of the reserve. The + # retry runs in a later invocation, and the second call answers. + client = fake_agentcore_client(invokes=[stall]) + config = config_with(client, retry_strategy=default_agentcore_retry_strategy) + started = time.monotonic() + + result = run(catching(config), {}, succeed_with("ok")) + + assert outcome(result) == {"result": "ok"} + invokes = client.requests("invoke_agent_runtime") + assert len(invokes) == 2 + assert invokes[0]["runtimeSessionId"] == invokes[1]["runtimeSessionId"] + # The stalled call blocks for WAIT seconds. The attempt ended after + # RESERVE_IN seconds, plus the 2-second retry delay. + assert time.monotonic() - started < WAIT / 2 + + +def test_fails_the_delivery_with_the_reserve_error_of_a_stalled_request( + run: Any, fake_agentcore_client: Any, near_reserve: None, stall: Any +) -> None: + client = fake_agentcore_client(invokes=[stall]) + + result = run(catching(config_with(client)), {}) + + error = outcome(result) + assert error["error"] == "AgentcoreDeliveryError" + assert ( + "InvokeAgentRuntime did not end before the invocation reserve" + in error["message"] + ) + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_a_stalled_stop_does_not_hide_the_recorded_result( + run: Any, fake_agentcore_client: Any, near_reserve: None, stall: Any +) -> None: + client = fake_agentcore_client(stops=[stall]) + started = time.monotonic() + + result = run(catching(config_with(client)), {}, succeed_with("done")) + + assert outcome(result) == {"result": "done"} + stop = result.get_context("review").get_step("review.stop") + assert stop.error is not None + assert "StopRuntimeSession did not end before the invocation reserve" in ( + stop.error.message or "" + ) + assert time.monotonic() - started < WAIT / 2 + + +def test_calls_without_a_deadline_outside_lambda() -> None: + context: Any = SimpleNamespace(lambda_context=None) + calls: list[str] = [] + + operation._call_before_reserve( # noqa: SLF001 + "StopRuntimeSession", + lambda: calls.append(threading.current_thread().name), + operation._remaining_seconds(context), # noqa: SLF001 + ) + + # Without a deadline, the call runs in the calling thread. + assert calls == [threading.current_thread().name] + + +def test_labels_every_operation_with_its_subtype(fake_agentcore_client: Any) -> None: + client = fake_agentcore_client() + runner = DurableFunctionTestRunner( + handler=catching(config_with(client)), poll_interval=0.05 + ) + try: + arn = runner.run_async(input="{}", execution_timeout=WAIT) + callback_id = runner.wait_for_callback( + arn, name="review.callback", timeout=WAIT + ) + runner.send_callback_success(callback_id, b'"done"') + result = runner.wait_for_result(arn, timeout=WAIT) + history = runner.get_execution_history(arn) + finally: + runner.close() + + assert outcome(result) == {"result": "done"} + expected = { + "review": "Agentcore", + "review.session": "AgentcoreSessionId", + "review.callback": "AgentcoreCallback", + "review.request": "AgentcoreRequest", + "review.stop": "AgentcoreStop", + } + context = result.get_context("review") + assert { + "review": context.sub_type, + **{child.name: child.sub_type for child in context.child_operations}, + } == expected + # The execution history carries the subtype on every event of each + # operation. + recorded: dict[str, set[str | None]] = {} + for event in history.events: + if event.name in expected: + recorded.setdefault(event.name, set()).add(event.sub_type) + assert recorded == {name: {sub_type} for name, sub_type in expected.items()} + assert {member.value for member in AgentcoreOperationSubType} == set( + expected.values() + ) + + +def stop_failure_logs(caplog: pytest.LogCaptureFixture) -> list[str]: + return [ + record.getMessage() + for record in caplog.records + if "StopRuntimeSession failed" in record.getMessage() + ] + + +def test_makes_no_call_with_less_than_11_seconds_left( + run: Any, + fake_agentcore_client: Any, + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, +) -> None: + # A function timeout of 3 seconds cannot fit the 10-second reserve and + # one call. So neither step calls the API, and the stop log does not + # suggest a missing permission. + monkeypatch.setattr(operation, "_remaining_seconds", lambda context: lambda: 3.0) + client = fake_agentcore_client() + + result = run(catching(config_with(client)), {}) + + error = outcome(result) + assert error["error"] == "AgentcoreDeliveryError" + assert "Raise the function timeout." in error["message"] + assert client.calls == [] + stop = result.get_context("review").get_step("review.stop") + assert stop.error is not None + assert "Raise the function timeout." in (stop.error.message or "") + [log] = stop_failure_logs(caplog) + assert "Raise the function timeout." in log + assert "permission" not in log + + +def test_names_the_permission_when_stop_is_denied( + run: Any, fake_agentcore_client: Any, caplog: pytest.LogCaptureFixture +) -> None: + client = fake_agentcore_client(stops=[agentcore_error("AccessDeniedException")]) + + result = run(catching(config_with(client)), {}, succeed_with("done")) + + assert outcome(result) == {"result": "done"} + [log] = stop_failure_logs(caplog) + assert "bedrock-agentcore:StopRuntimeSession" in log + + +def test_default_retry_strategy_retries_the_time_preflight() -> None: + error = MicrovmEndpointUnavailableError("Raise the function timeout.") + + assert default_agentcore_retry_strategy(error, 1).should_retry diff --git a/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_worker_composed_test.py b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_worker_composed_test.py new file mode 100644 index 000000000..121c927ce --- /dev/null +++ b/packages/aws-durable-execution-sdk-python-extras/tests/compute/agentcore/agentcore_worker_composed_test.py @@ -0,0 +1,227 @@ +# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. +# +# SPDX-License-Identifier: Apache-2.0 +"""The agentcore operation and the real AgentCore worker, end to end, in one +process. + +1. The durable function runs on the local test runner. +2. The fake ``InvokeAgentRuntime`` POSTs the payload to ``/invocations`` of a + real worker on loopback, with the session ID header, as AgentCore does. + It returns the worker's answer as the response stream. +3. The worker runs the job, and reports through a Lambda client double that + forwards each callback call to the runner. + +So the test covers the job document and the callback contract between the +two packages. +""" + +from __future__ import annotations + +import http.client +import json +from collections.abc import Callable, Iterator +from typing import Any + +import pytest +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.context import DurableContext +from aws_durable_execution_sdk_python.execution import ( + InvocationStatus, + durable_execution, +) +from aws_durable_execution_sdk_python.lambda_service import ErrorObject +from aws_durable_execution_sdk_python_compute_worker.agentcore import ( + AgentcoreJobContext, + AgentcoreWorker, + start_agentcore_worker, +) +from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner + +from aws_durable_execution_sdk_python_extras.compute.agentcore import ( + AgentcoreConfig, + AgentcoreError, + agentcore, +) + + +WAIT = 30 +ARN = "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/reviewer-abc123" + + +class RunnerCallbacks: + """A Lambda client for the worker that sends each callback call to the + local test runner.""" + + def __init__(self, runner: DurableFunctionTestRunner) -> None: + self._runner = runner + self.heartbeats = 0 + + def send_durable_execution_callback_success( + self, CallbackId: str, Result: bytes | None = None + ) -> dict[str, Any]: + self._runner.send_callback_success(CallbackId, Result) + return {} + + def send_durable_execution_callback_failure( + self, CallbackId: str, Error: dict[str, Any] + ) -> dict[str, Any]: + self._runner.send_callback_failure( + CallbackId, + ErrorObject( + message=Error.get("ErrorMessage"), + type=Error.get("ErrorType"), + data=None, + stack_trace=None, + ), + ) + return {} + + def send_durable_execution_callback_heartbeat( + self, CallbackId: str + ) -> dict[str, Any]: + self.heartbeats += 1 + self._runner.send_callback_heartbeat(CallbackId) + return {} + + +def invoke_worker(worker: AgentcoreWorker) -> Callable[[dict[str, Any]], bytes]: + """An InvokeAgentRuntime answer that POSTs the payload to ``worker``.""" + + def deliver(request: dict[str, Any]) -> bytes: + connection = http.client.HTTPConnection("127.0.0.1", worker.port, timeout=WAIT) + try: + connection.request( + "POST", + "/invocations", + body=request["payload"], + headers={ + "Content-Type": request["contentType"], + "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": request[ + "runtimeSessionId" + ], + }, + ) + response = connection.getresponse() + assert response.status == 200 + return response.read() + finally: + connection.close() + + return deliver + + +@pytest.fixture +def composed(fake_agentcore_client: Any) -> Iterator[Callable[..., Any]]: + """Run a job through the operation and a real worker.""" + closers: list[Callable[[], None]] = [] + + def _run( + job: Callable[[Any, AgentcoreJobContext], Any], + job_input: Any, + *, + deliveries: int = 1, + **fields: Any, + ) -> tuple[Any, Any, RunnerCallbacks]: + holder: dict[str, AgentcoreConfig] = {} + + @durable_execution + def handler(event: Any, context: DurableContext) -> Any: + try: + return { + "result": agentcore( + context, "review", job_input=event, config=holder["config"] + ) + } + except AgentcoreError as error: + return {"error": type(error).__name__, "message": error.message} + + runner = DurableFunctionTestRunner(handler=handler, poll_interval=0.05) + closers.append(runner.close) + callbacks = RunnerCallbacks(runner) + worker = start_agentcore_worker( + handler=job, + # The double has the three callback methods that the worker calls. + create_client=lambda region: callbacks, # type: ignore[arg-type, return-value] + heartbeat_interval_seconds=fields.pop("heartbeat_interval", None), + host="127.0.0.1", + port=0, + ) + closers.append(worker.close) + deliver = invoke_worker(worker) + + def deliver_times(request: dict[str, Any]) -> bytes: + answers = [deliver(request) for _ in range(deliveries)] + return answers[-1] + + client = fake_agentcore_client(invokes=[deliver_times]) + holder["config"] = AgentcoreConfig( + agent_runtime_arn=ARN, + timeout=Duration.from_minutes(5), + client=client, + **fields, + ) + arn = runner.run_async(input=json.dumps(job_input), execution_timeout=WAIT) + result = runner.wait_for_result(arn, timeout=WAIT) + assert worker.idle(WAIT) + assert result.status is InvocationStatus.SUCCEEDED, result.error + return json.loads(result.result or "null"), client, callbacks + + yield _run + for close in reversed(closers): + close() + + +def test_returns_the_result_of_the_job_in_the_worker(composed: Any) -> None: + def job(job_input: Any, context: AgentcoreJobContext) -> Any: + return {"sum": sum(job_input["numbers"]), "session": context.session_id} + + result, client, _ = composed(job, {"numbers": [1, 2, 3]}) + + [invoke] = client.requests("invoke_agent_runtime") + assert result == {"result": {"sum": 6, "session": invoke["runtimeSessionId"]}} + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_fails_with_the_error_type_that_the_worker_reports(composed: Any) -> None: + def job(job_input: Any, context: AgentcoreJobContext) -> Any: + msg = "lint failed" + raise ValueError(msg) + + result, client, _ = composed(job, {}) + + assert result == { + "error": "AgentcoreJobFailedError", + "message": 'AgentCore "review": the job failed (ValueError): lint failed', + } + assert len(client.requests("stop_runtime_session")) == 1 + + +def test_runs_a_job_that_arrives_twice_once(composed: Any) -> None: + runs: list[Any] = [] + + def job(job_input: Any, context: AgentcoreJobContext) -> Any: + runs.append(job_input) + return "done" + + result, _, _ = composed(job, {"n": 1}, deliveries=2) + + assert result == {"result": "done"} + assert runs == [{"n": 1}] + + +def test_sends_heartbeats_when_the_job_has_a_heartbeat_timeout( + composed: Any, +) -> None: + def job(job_input: Any, context: AgentcoreJobContext) -> Any: + context.cancelled.wait(0.3) + return "done" + + result, _, callbacks = composed( + job, + {}, + heartbeat_timeout=Duration.from_seconds(30), + heartbeat_interval=0.05, + ) + + assert result == {"result": "done"} + assert callbacks.heartbeats >= 1 diff --git a/packages/aws-durable-execution-sdk-python-extras/tests/conftest.py b/packages/aws-durable-execution-sdk-python-extras/tests/conftest.py index 880a39779..3e7b9c046 100644 --- a/packages/aws-durable-execution-sdk-python-extras/tests/conftest.py +++ b/packages/aws-durable-execution-sdk-python-extras/tests/conftest.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates. # # SPDX-License-Identifier: Apache-2.0 -"""Test doubles for the Lambda MicroVMs client.""" +"""Test doubles for the Lambda MicroVMs and the bedrock-agentcore clients.""" from __future__ import annotations @@ -136,6 +136,79 @@ def get_microvm(self, **request: Any) -> dict[str, Any]: return response +class FakeStream: + """The streaming body of an ``InvokeAgentRuntime`` answer.""" + + def __init__(self, data: bytes) -> None: + self._data = data + self.closed = False + + def read(self, amt: int | None = None) -> bytes: + data, self._data = ( + (self._data, b"") if amt is None else (self._data[:amt], self._data[amt:]) + ) + return data + + def close(self) -> None: + self.closed = True + + +class FakeAgentcoreClient: + """Records ``bedrock-agentcore`` API calls, and answers from scripted lists. + + ``invokes`` and ``stops`` hold the answers of ``invoke_agent_runtime`` + and ``stop_runtime_session``, in order. Each answer is an exception to + raise, a function that receives the request and returns the answer body, + bytes for the answer body, or ``None`` for ``{"accepted":true}``. After a + list ends, every call succeeds. + """ + + def __init__( + self, + *, + invokes: list[Any] | None = None, + stops: list[Any] | None = None, + ) -> None: + self.calls: list[tuple[str, dict[str, Any]]] = [] + self.streams: list[FakeStream] = [] + self._answers: dict[str, list[Any]] = { + "invoke_agent_runtime": list(invokes or []), + "stop_runtime_session": list(stops or []), + } + self._lock = threading.Lock() + + def requests(self, name: str) -> list[dict[str, Any]]: + with self._lock: + return [request for call, request in self.calls if call == name] + + def _answer(self, name: str, request: dict[str, Any]) -> Any: + with self._lock: + self.calls.append((name, request)) + answers = self._answers[name] + answer = answers.pop(0) if answers else None + if isinstance(answer, BaseException): + raise answer + if callable(answer): + return answer(request) + return answer + + def invoke_agent_runtime(self, **request: Any) -> dict[str, Any]: + body = self._answer("invoke_agent_runtime", request) + stream = FakeStream(b'{"accepted":true}' if body is None else body) + with self._lock: + self.streams.append(stream) + return {"response": stream, "statusCode": 200} + + def stop_runtime_session(self, **request: Any) -> dict[str, Any]: + self._answer("stop_runtime_session", request) + return {"statusCode": 200} + + +@pytest.fixture +def fake_agentcore_client() -> type[FakeAgentcoreClient]: + return FakeAgentcoreClient + + # The fakes are fixtures, not imports. Several packages in this repository # have a top-level ``tests`` package, so ``from tests.conftest import ...`` # could resolve to another package's tests. diff --git a/pyproject.toml b/pyproject.toml index 2011407cb..63fd318a8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -53,7 +53,7 @@ workspace.members = ["packages/*"] extra-dependencies = [ "mypy>=1.0.0", "pytest", - "boto3-stubs[lambda,lambda-microvms]", + "boto3-stubs[bedrock-agentcore,lambda,lambda-microvms]", "opentelemetry-sdk>=1.20.0", "opentelemetry-propagator-aws-xray", ] @@ -112,7 +112,7 @@ dependencies = [ "pytest-cov", "coverage[toml]", "mypy>=1.0.0", - "boto3-stubs[lambda,lambda-microvms]", + "boto3-stubs[bedrock-agentcore,lambda,lambda-microvms]", ] [tool.hatch.envs.dev-extras.scripts]