Skip to content

[Feature]: Harden createFileSystemSerdes publication and reference validation #860

Open

Description

@zhongkechen

What would you like?

Harden createFileSystemSerdes() so checkpoint file references remain immutable, integrity-checked, and bound to the durable payload that produced them.

The current implementation writes JSON to a deterministic <entityId>.json path with writeFile, then stores an unversioned {"file": ...} envelope. A later serialization for the same entity can overwrite content referenced by an older checkpoint, and deserialization trusts the file path carried in the envelope.

Previous fixes addressed entity-ID filename encoding and overflow sizing. The remaining desired guarantees are:

  • immutable file publication;
  • a versioned, self-identifying envelope;
  • durable-execution/entity ownership metadata;
  • content-integrity and path-safety validation;
  • continued support for ALWAYS, OVERFLOW, previews, URI, and hash modes.

Possible Implementation

  • Serialize once, compute a SHA-256 digest, and create a unique payload file with writeFile(..., { flag: "wx" }) or equivalent exclusive-create behavior.
  • Include an envelope marker/version, owner durable execution ARN, owner entity ID, payload type, file path, and content digest.
  • On deserialization, validate the recognized envelope, expected execution directory and filename, base-path containment, symbolic links, owner rules, and content digest.
  • Define explicit rules for safe incoming cross-execution references such as invoke input/results. Forwarded exceptions should be decoded and reserialized under the receiving child context when ownership changes.
  • Preserve compatibility by continuing to read the current legacy envelope format for an appropriate migration period.
  • Document storage lifecycle guidance because immutable publication can leave files that are no longer referenced by the latest checkpoint.
  • Extend the cloud coverage tracked by Add cloud-based integration tests for filesystem serdes #710 with immutable publication and ownership/integrity scenarios.

Is this a breaking change?

No. Existing configuration should remain compatible, and legacy envelopes can continue to be readable.

Does this require an RFC?

Yes.

Additional Context

The Java SDK filesystem SerDes work in aws/aws-durable-execution-sdk-java#648 uses immutable publication, versioned ownership metadata, content hashes, containment checks, and symbolic-link rejection. This issue requests equivalent guarantees adapted to the async TypeScript implementation.

Activity

  1. added
    enhancementNew feature or request
    pkg:sdkPackage: aws-durable-execution-sdk-js
    parityProvides parity with other language implementations of the SDK
    on Aug 25, 2026
  2. ParidelPooya commented on Nov 1, 2026

    @ParidelPooya
    Contributor

    Findings from a review of the current createFileSystemSerdes() behavior:

    Overwriting the same file is mostly fine. Each serialization for an entity writes <entityId>.json. Replay only reads the latest checkpoint, and the latest write holds the newest value, so last-write-wins matches what replay needs. For steps it's also safe: a step that dies before its checkpoint runs again and overwrites the file with a fresh result.

    There is one narrow gap, in waitForCondition. Every check serializes its state under the same stepId, so every check writes the same file:

    1. Check 1 writes state A and checkpoints. The checkpoint points at the file.
    2. Check 2 writes state B to the same file.
    3. The invocation dies before check 2's checkpoint lands.
    4. On replay the checkpoint still describes check 1, but the file now holds B. Check 2 re-runs starting from B instead of A.

    For polling-style state (for example "latest job status") this is harmless, because the check just runs again. It only matters when the check accumulates (for example count + 1), where one check can be applied twice. We haven't reproduced it yet; this is from reading the code.

    Suggested scope. Writing a unique file per serialization (for example with an exclusive create, as proposed) would close this gap. The other proposed guarantees (content digests, ownership metadata, symlink and containment checks) protect checkpoint data that is already trusted, so we'd treat them as separate, lower-priority hardening.

    Keeping this open to track the unique-file fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestparityProvides parity with other language implementations of the SDKpkg:sdkPackage: aws-durable-execution-sdk-js

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions