Repository navigation
fix(sdk): stop the invocation's work at its deadline and after termination #1202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
90158c2
test(sdk): show that an operation runs after the invocation terminated
ParidelPooya ce628df
fix(sdk): start no operation after the invocation terminated
ParidelPooya 8f88e0d
test(sdk): show that the invocation does not stop at its deadline
ParidelPooya 387d797
fix(sdk): terminate the invocation when its deadline passes
ParidelPooya d6c8d69
docs(sdk): describe the deadline termination for Lambda Managed Insta…
ParidelPooya 40d39f3
test(otel): report time left in two propagation test contexts
ParidelPooya 8924203
test(sdk): show three gaps in the deadline termination
ParidelPooya 0c19c72
fix(sdk): close three gaps in the deadline termination
ParidelPooya f04dece
fix(sdk): send queued checkpoints after a termination again
ParidelPooya fc02420
Merge branch 'main' into fix/sdk-no-operations-after-termination
ParidelPooya 637827c
docs(sdk): say that RETRYING does not promise a later invocation
ParidelPooya 2632c22
docs(sdk): tell plugins to end open operations on RETRYING
ParidelPooya 4d46660
Merge branch 'main' into fix/sdk-no-operations-after-termination
ParidelPooya 2bc45ab
Merge branch 'main' into fix/sdk-no-operations-after-termination
ParidelPooya 8e192fb
Merge remote-tracking branch 'origin/main' into fix/sdk-no-operations…
ParidelPooya 69b1d4b
test(sdk): show that a step result is serialized after the deadline
ParidelPooya 23bd407
fix(sdk): start no serialization after the invocation terminated
ParidelPooya 8810286
Merge branch 'main' into fix/sdk-no-operations-after-termination
ParidelPooya 529b9fe
Merge branch 'main' into fix/sdk-no-operations-after-termination
ParidelPooya File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
147 changes: 147 additions & 0 deletions
147
...execution-sdk-js/src/context/durable-context/operation-after-termination.composed.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| /** | ||
| * An operation that the handler starts after the invocation terminated must not run. | ||
| * | ||
| * Termination ends the invocation: the wrapper answers, and the checkpoint manager accepts no | ||
| * further checkpoint. The handler's own code can still run after that, for example when it | ||
| * awaits a timer or I/O outside any operation. On standard Lambda the execution environment | ||
| * freezes, so that code never resumes. On Lambda Managed Instances it does not freeze. A step | ||
| * that the handler starts then would run its function, and the service would never learn of it. | ||
| * | ||
| * Every test here holds a callback pending, waits outside any operation until the invocation | ||
| * suspends, and then starts one more operation. | ||
| */ | ||
| import { describe, it, expect } from "vitest"; | ||
| import { Context } from "aws-lambda"; | ||
|
|
||
| import { withDurableExecution } from "../../with-durable-execution"; | ||
| import { DurableExecutionInvocationInputWithClient } from "../../utils/durable-execution-invocation-input/durable-execution-invocation-input"; | ||
| import { hashId } from "../../utils/step-id-utils/step-id-utils"; | ||
| import { | ||
| DurableContext, | ||
| DurableExecutionClient, | ||
| InvocationStatus, | ||
| OperationSubType, | ||
| } from "../../types"; | ||
| import { | ||
| CheckpointDurableExecutionResponse, | ||
| GetDurableExecutionStateResponse, | ||
| OperationStatus, | ||
| OperationType, | ||
| WireOperation, | ||
| } from "../../types/wire"; | ||
| import { CHECKPOINT_TERMINATION_COOLDOWN_MS } from "../../utils/constants/constants"; | ||
|
|
||
| // Longer than the cooldown, so the invocation suspends while the handler waits. | ||
| const WAIT_OUTSIDE_OPERATION_MS = CHECKPOINT_TERMINATION_COOLDOWN_MS * 3 + 30; | ||
|
|
||
| const sleep = (ms: number): Promise<void> => | ||
| new Promise((resolve) => setTimeout(resolve, ms)); | ||
|
|
||
| const lambdaContext = { | ||
| awsRequestId: "request-1", | ||
| getRemainingTimeInMillis: () => 300_000, | ||
| } as unknown as Context; | ||
|
|
||
| const now = (): string => new Date().toISOString(); | ||
|
|
||
| const executionOp = { | ||
| Id: hashId("execution"), | ||
| Type: OperationType.EXECUTION, | ||
| Status: OperationStatus.STARTED, | ||
| StartTimestamp: now(), | ||
| ExecutionDetails: { InputPayload: "{}" }, | ||
| } as unknown as WireOperation; | ||
|
|
||
| /** The first root operation: a callback that was created and never completes. */ | ||
| const pendingCallbackOp = { | ||
| Id: hashId("1"), | ||
| Type: OperationType.CALLBACK, | ||
| SubType: OperationSubType.CALLBACK, | ||
| Name: "cb", | ||
| Status: OperationStatus.STARTED, | ||
| StartTimestamp: now(), | ||
| CallbackDetails: { CallbackId: "callback-id" }, | ||
| } as unknown as WireOperation; | ||
|
|
||
| const client: DurableExecutionClient = { | ||
| getExecutionState: async (): Promise<GetDurableExecutionStateResponse> => ({ | ||
| Operations: [], | ||
| NextMarker: undefined, | ||
| }), | ||
| checkpoint: async (): Promise<CheckpointDurableExecutionResponse> => ({ | ||
| CheckpointToken: "next-token", | ||
| NewExecutionState: undefined, | ||
| }), | ||
| }; | ||
|
|
||
| /** | ||
| * Runs `lateOperation` after the invocation suspended. Returns the invocation response and | ||
| * whether the late operation ran its function or settled. | ||
| */ | ||
| const run = async ( | ||
| lateOperation: (context: DurableContext, markRan: () => void) => unknown, | ||
| ): Promise<{ response: unknown; ran: boolean; settled: boolean }> => { | ||
| let ran = false; | ||
| let settled = false; | ||
| const response = await withDurableExecution( | ||
| async (_event: unknown, context: DurableContext) => { | ||
| const [callbackPromise] = await context.createCallback("cb"); | ||
| // Promise.all awaits the callback at once, so the callback is waiting while the | ||
| // other branch sleeps outside any operation. The invocation then suspends. | ||
| const [result] = await Promise.all([ | ||
| callbackPromise, | ||
| (async (): Promise<void> => { | ||
| await sleep(WAIT_OUTSIDE_OPERATION_MS); | ||
| await lateOperation(context, () => { | ||
| ran = true; | ||
| }); | ||
| settled = true; | ||
| })(), | ||
| ]); | ||
| return result; | ||
| }, | ||
| )( | ||
| new DurableExecutionInvocationInputWithClient( | ||
| { | ||
| DurableExecutionArn: | ||
| "arn:aws:lambda:us-east-1:123456789012:function:repro:$LATEST", | ||
| CheckpointToken: "token-1", | ||
| InitialExecutionState: { | ||
| Operations: [executionOp, pendingCallbackOp], | ||
| NextMarker: "", | ||
| }, | ||
| }, | ||
| client, | ||
| ), | ||
| lambdaContext, | ||
| ); | ||
| // The handler is still running. Give it time to start the late operation. | ||
| await sleep(WAIT_OUTSIDE_OPERATION_MS * 2); | ||
| return { response, ran, settled }; | ||
| }; | ||
|
|
||
| describe("an operation started after the invocation terminated", () => { | ||
| it("step: does not run the step function", async () => { | ||
| const { response, ran, settled } = await run((context, markRan) => | ||
| context.step("late", async () => { | ||
| markRan(); | ||
| }), | ||
| ); | ||
|
|
||
| expect(response).toEqual({ Status: InvocationStatus.PENDING }); | ||
| expect(ran).toBe(false); | ||
| expect(settled).toBe(false); | ||
| }); | ||
|
|
||
| it("runInChildContext: does not run the child function", async () => { | ||
| const { response, ran, settled } = await run((context, markRan) => | ||
| context.runInChildContext("late", async () => { | ||
| markRan(); | ||
| }), | ||
| ); | ||
|
|
||
| expect(response).toEqual({ Status: InvocationStatus.PENDING }); | ||
| expect(ran).toBe(false); | ||
| expect(settled).toBe(false); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
18 changes: 18 additions & 0 deletions
18
...urable-execution-sdk-js/src/errors/invocation-deadline-error/invocation-deadline-error.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| import { TerminationReason } from "../../termination-manager/types"; | ||
| import { UnrecoverableInvocationError } from "../unrecoverable-error/unrecoverable-error"; | ||
|
|
||
| /** | ||
| * The invocation's deadline passed before the handler finished. | ||
| * | ||
| * Thrown from the invocation, so it is an invocation error and Lambda retries the | ||
| * invocation. It does not fail the execution. Lambda has usually answered with its own | ||
| * timeout already, so this answer is then ignored. | ||
| */ | ||
| export class InvocationDeadlineError extends UnrecoverableInvocationError { | ||
| override name = "InvocationDeadlineError"; | ||
| readonly terminationReason = TerminationReason.INVOCATION_DEADLINE_PASSED; | ||
|
|
||
| constructor() { | ||
| super("Invocation deadline passed"); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.