Skip to content

v4: MemoryStorageClient's DatasetClient stores records by reference #3812

Closed

Description

@barjin
let data = {
    rand: 0
};

async function refreshData() {
    data.rand = Math.random();
}

async function pushAndRead({ storageClient }: { storageClient: StorageClient }) {
    const dataset = await Dataset.open(null, { storageClient });
    refreshData();
    await dataset.pushData(data);
    refreshData();
    await dataset.pushData(data);
    refreshData();
    await dataset.pushData(data);

    console.log((await dataset.getData()).items);
}

await pushAndRead({ storageClient: new MemoryStorageClient() }); 
// memory storage client will print the same number three times, as all the items refer to the same object
/// [
///  { rand: 0.22326410406394825 },
///  { rand: 0.22326410406394825 },
///  { rand: 0.22326410406394825 }
/// ]

await pushAndRead({ storageClient: new FileSystemStorageClient() });
// file system client takes snapshots of the passed objects
/// [
///  { rand: 0.6922433682025506 },
///  { rand: 0.33313558872226434 },
///  { rand: 0.6864574308371908 }
/// ]

DatasetClient implementations are inconsistent across different stock StorageClients.

MemoryStorageClient's Dataset will store the direct reference to the object instead of taking a JSON snapshot.

This can lead to unexpected mutations (see snippet above) or, e.g., unexpected GC consequences (see #3809).

Activity

  1. added this to the 4.0 milestone on Jul 2, 2026
  2. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jul 2, 2026
  3. georgekimaninjoroge commented on Jul 5, 2026

    @georgekimaninjoroge

    Is this open for external contributions? I'd be happy to take a look

  4. barjin commented on Jul 7, 2026

    @barjin
    MemberAuthor

    All of the Crawlee issues are open for all contributors - feel free to send a PR!

    Note that the problem described here is only present in the v4 branch - make sure to open the PR against this branch (not master) if you decide to work on this.

  5. added a commit that references this issue on Jul 7, 2026
    3ac3229
  6. AmariahAK commented on Jul 7, 2026

    @AmariahAK
    Contributor

    @barjin fixed it in this pr #3831

  7. barjin commented on Jul 10, 2026

    @barjin
    MemberAuthor

    Closed by #3831

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions