Repository navigation
refactor(sdk): create one plugin instance per invocation - #924
Merged
Merged
Conversation
ParidelPooya
had a problem deploying
to
ai-pr-review-runtime
September 17, 2026 13:47 — with
GitHub Actions
Failure
ParidelPooya
changed the base branch from
fix/insight-per-execution-export
to
main
September 17, 2026 17:14
The SDK creates one plugin instance per handler-module initialization, so a single Workflow Insight plugin serves every execution its environment hosts. Lambda Managed Instances makes concurrent executions in one environment routine. The export scheduler held one pending record for the whole plugin and overwrote it regardless of which execution the record belonged to. Measured on this package before the change, 20 trials per case: with N concurrent executions in on-complete mode exactly 2 terminal records were delivered and N-2 were lost (8 of 10 lost at N=10); in on-change mode 2 concurrent executions lost a terminal record in 20 of 20 trials. drain() returned successfully after exporting some other execution's record, so the loss was silent. Each execution now owns its pending record, its queue position and its drain signal, so coalescing happens only within one execution and drain(executionArn) returns only once that execution's own latest record has reached every exporter. Exporter calls stay serialized per plugin instance: at most one export at a time, and a flush is served by the same pump, so an exporter never sees flush() overlap export(). Also fixed, each found while reviewing the change above: - A synchronous throw from export(), render() or truncateRecord() escaped the fan-out before Promise.allSettled, hung the invocation to its timeout, left the scheduler unable to start another pump for any later execution, and produced an unhandled rejection. - The pump re-armed itself with a synchronous self-call, so a queue of failing fan-outs grew the stack one frame pair per execution, overflowed, and silently abandoned the tail. - A throw between closing an execution's scope and removing it left a closed scope in the map, muting that execution for the life of the environment. - Per-execution state survived a non-terminal (PENDING) invocation end, pinning the whole cached execution input until the environment died. State is now dropped at every invocation end and rebuilt from InvocationStartInfo. - An operation-change hook arriving after onInvocationEnd emitted a RUNNING record after the terminal one, reverting a finished execution for every exporter that upserts by execution ARN. - startTime trusted an un-normalized SDK value and fell back to now, so a resumed execution's durationMs covered only its last invocation. An Invalid Date now degrades one field instead of throwing out of record building. - InsightExporter.flush() had no documented contract. It now states the cadence, the exclusivity guarantee, that a flush may cover other executions' records, and how failures are handled. No public type or exported symbol changed. Record shape, emit modes, sampling, per-exporter truncation and the default exporter are unchanged.
ParidelPooya
force-pushed
the
refactor/per-invocation-plugin-instances
branch
from
September 17, 2026 17:20
05b9d11 to
acad423
Compare
A plugin instance used to live as long as the execution environment while serving every execution that landed on it. Under Lambda Managed Instances several executions run concurrently in one environment, so every plugin had to key its own state by execution ARN and clean that map up itself. Getting that wrong loses telemetry silently, and both bundled plugins had got it wrong. The SDK now builds one plugin per invocation and drops it when the invocation ends. `plugins` takes factories, not instances: plugins: [(info) => new MyPlugin(sharedExporter, info.executionArn)] A factory is a bare function taking `InvocationInfo` and returning a plugin. Errors it throws are contained exactly like errors thrown by a hook: the invocation proceeds without that plugin. Environment-lifetime state stays in the closure or in a factory object; per-invocation state becomes ordinary instance fields. This deletes the provider path rather than adding a parallel one. While an instance path exists a plugin cannot delete its ARN-keyed map, which is the entire point of the change. Removed: `DurableInstrumentationPluginProvider`, `DurableInstrumentationPluginType`, and `DURABLE_INSTRUMENTATION_PLUGIN_API_VERSION`. Added: `DurableInstrumentationPluginFactory`. `durableExecutionPluginProvider` is now a factory, so environment-based loading works unchanged. Workflow Insight now holds no ARN-keyed structure at all. It also drops `wrapInvocation` and drains from `onInvocationEnd`, which the SDK awaits at every exit including the config-error path that `wrapInvocation` missed. The OTel plugin resolves the tracer provider once per environment and keeps each invocation's spans in plain fields, so `resetInvocationState` is gone. BREAKING CHANGE: `plugins` accepts factories instead of plugin instances, and the provider interface is removed. Pass `(info) => new MyPlugin()` where you passed `new MyPlugin()`, and export a factory where you exported a provider. Not migrating no longer degrades quietly: a plugin instance, or the plugin class itself, in `plugins` now fails every invocation with `PluginLoadError` at handler initialization, rather than leaving the plugin silently absent.
…p failure Review findings on the per-invocation plugin contract. Both blocking items share a shape: a failure that used to be loud became silent. A JavaScript caller who does not migrate passes a plugin instance in `plugins`. The SDK called it as a factory, the call threw `TypeError`, and the containment in `createInvocationPluginRunner` discarded the entry and wrote nothing, so the plugin was absent for the life of the environment with no signal at all. The loader already validated that an environment-selected provider is callable and raised `PluginLoadError`, so the two paths disagreed. `loadConfiguredPlugins` now runs the same check over the explicit entries, naming the offending position. A factory that throws when called stays contained, because that is the documented contract; an entry that is not callable at all is a packaging mistake that would otherwise be rediscovered and swallowed on every invocation. The OTel environment is built lazily inside the per-invocation factory, so a throwing `tracerProviderFactory` now throws inside an invocation instead of at Lambda init. Containment is right, but the attempt repeated: the customer's factory ran again on every invocation for the life of the environment, and a factory that builds a span processor before throwing leaked one per invocation. Creation is now attempted at most once, the failure is logged once, and later invocations are refused with the remembered error. A working factory is still resolved exactly once. Also from the review: `replayLinks` dropped a guard that became unreachable when `initialOperationLink` stopped returning `undefined`; `describeValue` reported an array as "an object" and would have said "a undefined"; and the `flush()` contract now documents that a slow `export` multiplies by environment concurrency, with every concurrent end paying the whole product, since the pump exports every awaited record before the shared flush turn.
A caller migrating from the earlier contract is most likely to pass the class,
because `plugins: [MyPlugin]` was once close enough to correct to look right.
`typeof MyPlugin === "function"`, so the callable check added in the previous
commit does not see it. Calling it throws "Class constructor cannot be invoked
without 'new'" on every invocation, and that throw is contained the way any
plugin failure is contained, so the plugin is absent for the life of the
execution environment with nothing said about why.
I had claimed this needed a per-invocation check to detect. That was wrong: the
source text of a class begins with the `class` keyword and nothing else callable
does, so `Function.prototype.toString.call(entry)` settles it at load time. Both
plugin paths now reject it there, and the message says what to pass instead.
The keyword must be followed by whitespace, `{`, or a comment start. Without
that, a shorthand method named `class` stringifies as `class(info) {}` and one
named `classify` as `classify(info) {}`, and both would be mistaken for a class.
`toString` is called through `Function.prototype` rather than through the value,
because a function can carry its own `toString`.
Measured while writing this: a class declaration, a named or anonymous class
expression, `class{}` with no space, a subclass, and a class with a TS 5
decorator all keep the keyword, as does esbuild minifying to ES2022. Downlevelling
to ES5 does not — TypeScript and esbuild both emit `function PlainPlugin() {}` —
so a class transpiled that far is still missed. Such a function called without
`new` returns `undefined` rather than throwing, and the runner already skips a
factory that hands back nothing, so containment remains the backstop for exactly
that case. Nothing further is attempted: the heuristics that would catch it,
inspecting `prototype` descriptors or method enumerability, also reject ordinary
factory functions, and a false rejection at load time would fail an invocation
that would otherwise have worked.
ParidelPooya
force-pushed
the
refactor/per-invocation-plugin-instances
branch
from
September 17, 2026 20:15
e040d90 to
d61acce
Compare
ParidelPooya
had a problem deploying
to
ai-pr-review-runtime
September 17, 2026 20:44 — with
GitHub Actions
Failure
Two findings the Python reviewers raised, checked here and both real. The first is the version set. Both plugin packages declared a peer range that admits a core without the factory contract: otel `>=2.4.0` and insight `>=2.1.0`, against a core at `2.4.0`. This branch deletes the provider contract, so npm accepts otel with core `2.4.0` and the handler then fails at initialization. `RELEASING.md:33` puts version numbers in the pull request rather than in the release workflow, so this commit sets them: core to `3.0.0` because `plugins` changed element type from instances to factories, otel to `2.0.0` because its plugin classes are no longer exported and factory builders replace them, and both peer ranges to `>=3.0.0`. Insight goes to `0.1.0-alpha.1` rather than `0.2.0-alpha.0`. Only `0.1.0-alpha.0` has ever been published, so a minor bump would claim a released `0.1.0` that does not exist. The prerelease counter is the smallest step that is greater than the published version, which the publish script requires. The testing package moves for a consequential reason rather than a chosen one. Its range was `>=1.0.1 <=2.4.0`, whose upper bound excludes the core built here, and its own test asserts the core satisfies that range. It references the plugin contract nowhere, so core `3.0.0` is compatible with it and widening the bound is correct. Its version follows, because the publish script skips a package whose version is unchanged and the widened range would never reach a consumer. The existing peer check covered the testing package only. It now scans every workspace manifest, and for a package that consumes the factory contract it requires the declared lower bound to name at least the core major built here. Which packages consume the contract is derived from their sources rather than from a list, because a list is what gets left behind. The second finding is the reentrancy defect the Python plugin had, and my earlier reading that this plugin was safe was wrong. I judged that `onOperationChange` builds and schedules in one expression with no `await` between the `closed` check and the `schedule` call, so nothing could interleave. Argument expressions evaluate before the call they feed, so `buildRecord` completes before `schedule` receives its result, and `buildRecord` calls customer code: the input and output content transforms, a per-operation result override, and any accessor on a customer-thrown error. A synchronous re-entry from that code interleaves without an `await`. Measured, with a content transform re-entering `onOperationChange` while the outer frame held the older snapshot: the exporter received `[['A','B'], ['A']]`, so an exporter that upserts by execution ARN stores the older snapshot. Measured with the same re-entry into `onInvocationEnd`: the exporter received a terminal record and then a stale RUNNING record for a finished execution, which is exactly what the `closed` flag exists to prevent and cannot, because the outer hook passed its check before the nested hook set it. Each invocation now carries a build revision. Building takes the revision, and the record is scheduled only while that revision is still current, so a superseded snapshot is dropped instead of overwriting a newer one. A record is a complete snapshot, so dropping a superseded one loses nothing. That is the same property that makes the scheduler's per-execution coalescing sound. One limit on the second finding, stated because it bounds the severity rather than the correctness. Reaching the plugin instance from a transform requires the customer to capture it from their own wrapping factory, which the new contract permits but does not require. So the reordering is measured and real, and its reachability from an unmodified handler is not established.
A reviewer asked for the registration type to be an object with a method rather than a bare callable, and the reason is future extensibility rather than taste. A function type has no member to add anything to. So adding a process-level hook later -- a flush when the execution environment shuts down, for example -- would have to change the registration type from function to object, which is a second breaking change on the same public surface. An object with one method can gain an optional second member additively, so doing this now costs one break instead of two. `DurableInstrumentationPluginFactory` is therefore an interface declaring `createPlugin(info)`, not `(info: InvocationInfo) => Plugin`. The type's name and the returned type's name are both unchanged, so no identifier changes meaning. The method is `createPlugin` rather than `newInvocation` because what it returns is still called a plugin, so the verb and the noun agree. This is the shape the Java SDK already had, as `DurableExecutionPluginFactory.createPlugin(InvocationInfo)`, so the three SDKs now describe one contract. The generic parameter is kept because it is used: the OTel builders return `DurableInstrumentationPluginFactory<InvocationOtelPlugin>` and `<ExecutionOtelPlugin>`, and without it a caller could no longer see which plugin type they get back. An interface method position is covariant in its return type the same way the function type was, so nothing weakened. The loader's shape test replaces `typeof value === "function"` with a property lookup for a callable `createPlugin`. It is a lookup rather than an own-property check, so a factory written as a class instance, which carries the method on its prototype, is accepted -- the runtime check has to accept exactly what the type accepts. A function carrying a `createPlugin` property is accepted for the same reason. The value is never called. One check became redundant and is deleted. An earlier commit in this branch added a load-time heuristic that rejected a class passed where a factory belongs, reading `Function.prototype.toString` for the `class` keyword, because `typeof MyPlugin === "function"` meant a class passed the callable test. A class carries no `createPlugin`, so the shape test rejects it directly. Deleting the heuristic also retires its documented limitation: a class transpiled to an ES5 `function` did not stringify as `class` and so escaped it, and that case is now caught like any other. Its tests move to the new rejection path rather than being removed, minus three that only pinned the regex. Migrated with it: both bundled builders, which now return the object; the three dynamic-plugin-layer provider fixtures, which exported a bare function and were genuinely broken rather than merely imprecise; the cjs and esm integration tests that consume them; 18 conformance plugin handlers, 19 factories; and the prose in two READMEs and four insight design documents. No handler's observable behaviour changed, and that was established from the diff rather than assumed. `git diff -w` over the conformance package shows only wrapper lines added or removed, so every factory body is byte-identical apart from indentation. No added line contains a backtick, so no template literal was re-indented. No `plugins:` array was edited, so registration order is preserved in both handlers that register two. The conformance and integration packages resolve `@aws/*` through a worktree-root symlink into a different clone, which is why an earlier pass could not verify them. Package-local symlinks were added under `node_modules` so they compile against this worktree; `node_modules` is gitignored, so none of that is committed.
onInvocationStart set tracingEnabled from ensureTracingEnabled at the top, then called the customer-supplied contextExtractor, and only assigned executionTraceId afterwards. An extractor that throws therefore left the instance enabled with no trace identity: the plugin runner contains the rejected hook, the invocation continues, and every later hook passes its tracingEnabled check. A cross-invocation replay or continuation span then linked against an empty trace ID. This PR removed the guard that used to catch that, on the reasoning that the link is always constructible. Its span ID is; its trace ID is not. The flag is now set last, so it means what every later hook reads it to mean: this invocation's trace identity is resolved.
The pump claims the pending flush resolvers with splice(0) before it front-loads the records other ends are waiting on. From that point they are not in flushWaiters, so nothing else can find them and no later pump serves them. The front-load await sat outside the try that releases them, and exportPending is a try/finally with no catch, so a synchronous throw from its fan-out escaped and stranded every waiter. Because onInvocationEnd is awaited before the Lambda response, those invocations hung to the function timeout. The claim and the release are now the same critical section. A flush that did not run is the same answer to a waiter as a flush that failed, which is what the per-exporter containment in flushAll already establishes. A comment claiming that exporters.map runs inside the pump's guarded region is corrected: it runs inside a finally, not a catch.
Both plugin packages declared >=3.0.0 on the core with no ceiling. That is the shape of the defect this PR already fixed once: otel declared >=2.4.0 against a core whose 2.4.0 could not load the factory contract, npm accepted it, and the handler failed at initialization. The next core major that changes the plugin contract reproduces it exactly. Both are now >=3.0.0 <4.0.0, and the monorepo guard asserts a contract-bound range rejects the next major rather than only accepting the current one. The testing package had the mirror problem: >=1.0.1 <=3.0.0 excludes core 3.0.1, so the first core patch would have put its claim out of date and its own guard red, for a package that touches the plugin contract nowhere. It is now >=1.0.1 <4.0.0, and a new case asserts every declared range admits the next core patch.
The lint job checks formatting; the local check I ran filtered to error-level diagnostics, which excludes it.
yaythomas
reviewed
Sep 29, 2026
Retain core SDK 3.0.0 and the plugin-compatible peer ranges while incorporating main's SDK 2.5.0 and testing SDK 1.2.0 changes. Advance the testing package to 1.2.1 and synchronize lockfile peer metadata. Update the plugin-lifetime checkpoint mock for the new token-revocation and execution-result methods introduced on main. Validation: full build, repository lint checks, source and conformance type checks, core SDK (1346), testing SDK (1227; 3 skipped), OTel (360), Insight (108), OTel conformance handlers (35), and CJS/ESM integration.
Keep the core SDK at 3.0.0 and preserve the plugin-compatible peer ranges. Advance the testing SDK to 1.3.1 after main moved to 1.3.0, with the matching lockfile version.
This was referenced Oct 2, 2026
2 tasks done
This comment has been minimized.
This comment has been minimized.
zhongkechen
previously approved these changes
Oct 19, 2026
zhongkechen
previously approved these changes
Oct 20, 2026
This comment has been minimized.
This comment has been minimized.
2 of 4 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Codex AI reviewNo actionable findings in the PR changes. Residual test risk: tests and cloud integration workflows were not executed during this read-only review. Reviewed commit |
yaythomas
approved these changes
Oct 21, 2026
This was referenced Oct 21, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The shared LocalDurableTestRunner fix landed in #1020. This PR targets
mainand prepares Core 3.0 / OTel 2.0 with per-invocation plugin factories. The renamed Durable Execution Telemetry exporter retains upstream's0.1.0-alpha.0version and adopts the matching Core 3 contract.Issue Link, if available
Refs #950. Related: #953, #954, #955, #956, #959, #960, #962, and #964. Business lifecycle cancellation and deadline handling remain tracked in #927/#1014.
Description
Handler-scoped plugin instances can mix concurrent Lambda Managed Instances executions. SDK 3.x accepts
createPlugin(info)factories and constructs one plugin instance per invocation. Spans, operation maps, clocks, execution identity, and buffers belong to that invocation; providers, exporters, and environment initialization remain shared. Declared legacy v1 providers fail before the factory or handler runs. Factories may return null/undefined to skip an invocation; bundled factories retain concrete return types.Both OTel views preserve bounded finalization: synchronous child completion and parent finalization, current checkpoint ancestry, terminal ERROR status without invented exceptions, public global-provider delegate flushing, and one
flushTimeoutMillisbudget across all waits. Hooks freeze before the final provider wait. Closing one instance does not close another invocation; attempts admitted during the open drain phase retain their identity and are included in the final bounded batch. The Telemetry exporter likewise keeps invocation-local buffers and shared serialized exports.The change includes invocation-local X-Ray carriers, replay export suppression, fresh operation-update visibility, exclusive OTel views, and chained-invoke propagation through each invocation's factory instance. Carrier availability remains authoritative even when its value is empty, undefined, null, or invalid. Shared-LMI examples require a real runtime carrier. Initialization failure closes any known Invocation span best effort, leaves that instance inert, and does not shut down the shared provider.
Physical span boundaries use invocation-local wall/monotonic correlation and
HrTime; supplied backend Dates remain authoritative. Missing logical boundaries use millisecond precision, and a missing deferred start is bounded by a known completion. Already-recording attempts cannot retroactively move their start across arbitrary backend clock skew.The integration with upstream's Telemetry rename updates package names, imports, examples, and documentation while retaining the major factory API and invocation isolation. All 26 conformance handlers use factory registration. The reusable workflow discovers the pinned catalog and no longer receives the unsupported
case_countinput.Late completions during the open drain phase can outlive already-ended enclosing spans. Updates after finalization, exporter failure, or a frozen runtime have no delivery guarantee. No export acknowledgment is persisted, and the OTel factory does not cancel detached business work.
Testing
Validation on
65a3a036, including upstream5f0c707d:skipLibCheck: false. Both loaders passed concurrent real public-wrapper invocations and completed-step replay with a fake transport. Verified invocation isolation, serialized exports, final flush, and renamed exports/discriminator; no runtime network calls occurred.No AWS cloud deployment was run locally for this merge.
GitHub checks and workflows passed on
65a3a036, including Build, Core and OTel conformance, CodeQL, and AI review.Demo/Screenshots
Not applicable; public-handler, replay, installed-package, and concurrent-invocation regressions cover the behavior.
Checklist
65a3a036