Skip to content

feat(extras): add experimental decider and llm operations - #928

Merged
ParidelPooya merged 5 commits into
mainfrom
feat/extras-ai
Nov 13, 2026
Merged

ParidelPooya merged 5 commits into
mainfrom
feat/extras-ai

Conversation

@ParidelPooya

Copy link
Copy Markdown
Contributor

Python counterpart of the JavaScript /ai subpath (aws/aws-durable-execution-sdk-js#1160, with the transports from aws/aws-durable-execution-sdk-js#1213). Draft: the JavaScript work waits for Bedrock to support decider models, so this one waits too.

What it adds

aws_durable_execution_sdk_python_extras.ai. The module and every export are marked experimental.

Operation Step name Subtype Returns
decider().check decider.check DeciderCheck bool
decider().classify decider.classify DeciderClassify one choice
decider().label decider.label DeciderLabel the choices that apply
decider().score decider.score DeciderScore 0 to 1
llm().ask llm.ask LlmAsk text

Step names and subtypes match the JavaScript SDK.

Where the decision model runs

Every Strands Decider host takes the same request. So decider() takes endpoint or one transport:

Host Use AWS call Permission
Any server endpoint= plain HTTP none
SageMaker transport=sagemaker_transport(endpoint_name) InvokeEndpoint sagemaker:InvokeEndpoint
Lambda transport=lambda_transport(function_name, qualifier=) Invoke lambda:InvokeFunction
AgentCore Runtime transport=agentcore_transport(runtime_arn) InvokeAgentRuntime bedrock-agentcore:InvokeAgentRuntime

boto3 is already a dependency of this package. So the transports need no new dependency.

Python-specific behavior

  • Every error extends AiError, which extends DurableOperationError and is registered with the SDK. So the call raises the same class on the first run and on replay. DeciderError.status and LlmStoppedError.stop_reason are stored in the error data, so they survive replay. Any other step failure stays a StepError.
  • boto3 and urllib cannot cancel a call. So each request runs in a daemon thread, and the step stops waiting 10 seconds before the invocation ends, with AiTimeoutError. The MicroVM operations use the same pattern.
  • An invalid argument raises TypeError or ValueError before any step starts.

Testing

  • pytest on the extras package: 563 passed (127 new). The new module has 97% line coverage.
  • The new tests run on the local test runner. They cover each operation, each transport with a fake client and with botocore Stubber, the error mapping, replay of each error class, the deadline, and a local HTTP server for endpoint.
  • mypy on the extras package and hatch fmt --linter --check pass. hatch fmt --check reports one file, tests/compute/microvm/aws_errors_test.py, which this PR does not change.
  • Not tested against a real SageMaker endpoint, Lambda function, AgentCore runtime, or Bedrock model. There is no e2e test yet.

Reference: packages/aws-durable-execution-sdk-python-extras/docs/ai.md.

The JavaScript extras package has an /ai subpath with decider() and
llm(). Python had no counterpart. This commit adds
aws_durable_execution_sdk_python_extras.ai with the same operations,
step names, subtypes and errors.

decider() returns check, classify, label and score. Each one asks a
Strands Decider one typed question in one durable step. A Strands
Decider can run on a server, on SageMaker, on Lambda, or on AgentCore
Runtime. Every host takes the same request body. So decider() takes
either endpoint, for plain HTTP, or a transport:

- sagemaker_transport(endpoint_name) calls InvokeEndpoint.
- lambda_transport(function_name, qualifier=) calls Invoke, with the
  request as the event.
- agentcore_transport(runtime_arn, qualifier=) calls InvokeAgentRuntime.

boto3 is already a dependency of this package. So the transports need
no new dependency.

llm() returns ask, which calls Bedrock Converse in one durable step.

Python-specific behavior:

- Every error extends AiError, which extends DurableOperationError and
  is registered with the SDK. So the call raises the same class on the
  first run and on replay. DeciderError.status and
  LlmStoppedError.stop_reason are stored in the error data, so they
  survive replay. Any other step failure stays a StepError.
- boto3 and urllib cannot cancel a call. So each request runs in a
  daemon thread, and the step stops waiting 10 seconds before the
  invocation ends, with AiTimeoutError. This is the pattern that the
  MicroVM operations use.
- An invalid argument raises TypeError or ValueError before any step
  starts.

The module and every export are marked experimental.
Fixes for hln33's review.

- check and label now reject a probability outside 0 to 1 with
  DeciderInvalidResponseError. A value within 0.01 of the range is
  clamped, as score does.
- decider() raises TypeError when headers is set with transport. The
  headers were dropped without an error.
- DeciderError keeps status, and LlmStoppedError keeps stop_reason, when
  data is also given. A dropped status read as unreachable and was
  retried.
- New tests check that a replay returns the checkpointed answer without
  calling the model, for decider and llm.
@ParidelPooya
ParidelPooya marked this pull request as ready for review November 13, 2026 18:27
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 13, 2026 18:27 — with GitHub Actions Active
@ParidelPooya
ParidelPooya merged commit 8fbb245 into main Nov 13, 2026
16 checks passed
@ParidelPooya
ParidelPooya deleted the feat/extras-ai branch November 13, 2026 18:28
ParidelPooya added a commit to aws/aws-durable-execution-sdk-js that referenced this pull request Nov 16, 2026
Brings the parked `/ai` work to main: #1160 (`decider()` and `llm()`)
and #1213 (SageMaker and Lambda transports), merged with current main,
plus fixes for the open review threads on #1213.

## What it adds

`@aws/durable-execution-sdk-js-extras/ai`. Every export is marked
`@experimental`.

| Operation | Step name | Subtype | Returns |
|---|---|---|---|
| `decider().check` | `decider.check` | `DeciderCheck` | `boolean` |
| `decider().classify` | `decider.classify` | `DeciderClassify` | one
choice |
| `decider().label` | `decider.label` | `DeciderLabel` | the choices
that apply |
| `decider().score` | `decider.score` | `DeciderScore` | 0 to 1 |
| `llm().ask` | `llm.ask` | `LlmAsk` | text |

| Decision model host | Use | Permission |
|---|---|---|
| Any server | `endpoint` | none |
| SageMaker | `transport: sageMakerTransport({ endpointName })` |
`sagemaker:InvokeEndpoint` |
| Lambda | `transport: lambdaTransport({ functionName, qualifier })` |
`lambda:InvokeFunction` |
| AgentCore Runtime | `transport: agentCoreTransport({ runtimeArn })` |
`bedrock-agentcore:InvokeAgentRuntime` |

The four AWS SDK clients are optional peer dependencies, loaded on first
use. Reference:
`packages/aws-durable-execution-sdk-js-extras/docs/ai.md`.

## Fixes on top of #1213

1. **esbuild bundles.** esbuild leaves a missing package to run time
only when `.catch()` sits directly on the literal `import()`. The
`lazySdk` helper in #1213 moved it into the helper. So bundling `/ai`
with esbuild failed unless all four clients were installed. Each
`import()` now has its own `.catch(sdkLoadError(...))`. `bundle.test.ts`
bundles a copy of `src/ai` where no client can be resolved. It fails
with the #1213 code and passes now.
2. **One default-client cache.** `lazyClient()` replaces the three
copies (hln33).
3. **Default client test.** Each transport creates one default client on
first use and reuses it.
4. **README typo** (hln33), now in `docs/ai.md`.

## Changes from merging main

- The ai tests now run on vitest.
- The AI reference moved from the README to `docs/ai.md`, as main did
for `docs/microvm.md`. The README links to it.
- `package-dependencies.test.ts` lists the four ai clients. It checks
that the package root and `/ai` load without them.

## Testing

- Extras `npm test` (tsc and vitest): 514 passed, 1 skipped (the live
test, which needs `AI_LIVE=1`).
- `npm run build` and `npm run lint` pass.
- Not tested against a real SageMaker endpoint or Lambda function. There
is no example in the examples package, because an example needs a
deployed decision model.

Python counterpart: aws/aws-durable-execution-sdk-python#928.

## Design note

**Problem.** A durable function often needs one typed decision (yes or
no, one of N, a score) or one short text answer from a model. Calling a
model inside `context.step` by hand repeats the same work in every
function: prompt shape, reply parsing, retries, the invocation deadline,
and checkpoint size.

**Cause.** The core SDK gives steps, not model calls. No existing
library gives model calls that are durable steps on Lambda. Typed-answer
libraries are not durable. Durable platforms ship model layers only for
their own runtimes.

**Options considered.**

| Option | Result |
|---|---|
| 1. Document a `context.step` + client pattern only | No new API. Every
function repeats parsing, retries and the deadline. |
| 2. One `ai()` namespace with all operations | One import. A decision
model and a text model share one config, so a default model is
ambiguous. |
| 3. `decider()` and `llm()` factories under `/ai` (chosen) | Config is
set once per factory. Two deciders can differ. Fakes are easy in tests.
|
| 4. One `decider()` config key per host (`agentCore`, `sageMaker`, ...)
| Each new host changes `DeciderConfig`. |
| 5. `endpoint` or one `transport` (chosen) | A new host, such as
Bedrock later, is one new export. `decider()` does not change. |

**Removed before release.** The `agentCore` config key (now
`agentCoreTransport`), and `AgentCoreInvokeClient` (now `AwsSdkClient`).
Neither was released.

**Defaults.** `llm()` uses a pinned Bedrock model ID, moved forward in
later releases. `decider()` has no default host, because no managed host
exists yet. It uses `STRANDS_DECIDER_ENDPOINT` when no option is set.
Step names and subtypes are fixed, because a change breaks executions in
flight.

**Experimental.** Every export is `@experimental`.

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 24be6dae Deployed Nov 13, 2026 by ParidelPooya via notify / Generate notification summary #416
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants