Skip to content

fix(memory-storage): store JSON snapshots in DatasetClient.pushData() - #3831

Merged
barjin merged 1 commit into
apify:v4from
AmariahAK:v4
Jul 9, 2026
Merged

barjin merged 1 commit into
apify:v4from
AmariahAK:v4

Conversation

@AmariahAK

Copy link
Copy Markdown
Contributor

PR Description

What was the issue

MemoryStorageClient's DatasetClient.pushData() stored the raw object reference into its internal Map rather than taking an independent deep copy. Later mutations to the original object were reflected in every stored entry — pushing the same mutable object multiple times (mutating between pushes) resulted in all items holding the same final value. This was inconsistent with FileSystemStorageClient, which serializes to disk on each push and therefore stores independent snapshots.

Where was the issue

packages/core/src/memory-storage/resource-clients/dataset.ts:127

this.datasetEntries.set(idx, entry);

This line stored the direct reference — no copy, no serialization.

How was it fixed

Store JSON.parse(JSON.stringify(entry)) instead of the raw object reference so that each push produces an independent deep copy. This makes MemoryStorageClient consistent with FileSystemStorageClient, which already stores independent snapshots.

this.datasetEntries.set(idx, JSON.parse(JSON.stringify(entry)) as Data);

Why this method over alternatives

  • JSON.parse(JSON.stringify()) is the canonical JavaScript deep-clone for JSON-serializable data — no dependencies, no overhead
  • The Dataset frontend already enforces JSON serializability via assertJsonSerializable before data reaches the client, so this clone is guaranteed to succeed for every valid item
  • structuredClone() handles types (Date, Map, ArrayBuffer, etc.) that dataset items can never contain (the validation gate rejects them), making it unnecessary overhead
  • No external clone library needed — zero new dependencies

How to test locally

pnpm test -- test/core/storages/dataset.test.ts

The added regression test "stores independent snapshots, not object references" pushes a mutable object three times with mutations between pushes, then asserts items have distinct values and are not the same reference.

Reproduction from the issue:

let data = { rand: 0 };
const refresh = () => { data.rand = Math.random(); };

const client = new MemoryStorageClient();
const dataset = await Dataset.open(null, { storageClient: client });
refresh(); await dataset.pushData(data);
refresh(); await dataset.pushData(data);
refresh(); await dataset.pushData(data);

const { items } = await dataset.getData();
// All three should now have different `rand` values — independent snapshots
console.log(items.map(i => i.rand));

Fixes #3812

Store JSON.parse(JSON.stringify(entry)) instead of the raw object
reference so that each push produces an independent deep copy. This
makes MemoryStorageClient consistent with FileSystemStorageClient,
which already stores independent snapshots.

Fixes apify#3812

Co-authored-by: atlarix-agent <agent@atlarix.dev>
@AmariahAK

This comment was marked as spam.

@barjin
barjin merged commit 9e87aa9 into apify:v4 Jul 9, 2026
8 checks passed
barjin pushed a commit that referenced this pull request Jul 20, 2026
…)` (#3831)

Serializes the passed data in `pushData` calls before storing it.
B4nan pushed a commit that referenced this pull request Aug 12, 2026
…)` (#3831)

Serializes the passed data in `pushData` calls before storing it.
B4nan pushed a commit that referenced this pull request Aug 18, 2026
…)` (#3831)

Serializes the passed data in `pushData` calls before storing it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants