Skip to content

fix(memory-storage): store snapshot in DatasetMemoryEntry.update - #3826

Closed
AmariahAK wants to merge 1 commit into
apify:masterfrom
AmariahAK:master
Closed

AmariahAK wants to merge 1 commit into
apify:masterfrom
AmariahAK:master

Conversation

@AmariahAK

@AmariahAK AmariahAK commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

DatasetMemoryEntry.update() stores a JSON snapshot of the provided
data instead of the direct object reference. Previously, when the same
mutable object was pushed multiple times with mutations between pushes,
all stored entries reflected the final mutated state because they
shared the same reference.
This is inconsistent with the file-system backed implementation
(DatasetFileSystemEntry), which serializes to JSON on write and
parses on read — naturally creating independent copies.
The fix changes this.data = data to
this.data = JSON.parse(JSON.stringify(data)) in
packages/memory-storage/src/fs/dataset/memory.ts.
A test (dataset-snapshot.test.ts) verifies that each push creates an
independent snapshot by pushing a mutable object three times with
mutations between pushes and asserting each stored entry preserves its
push-time value.
Related: #3809 (stale references held in memory may prevent GC — the
snapshot approach helps by breaking reference chains, though that issue
is not fully resolved by this change alone).
Closes #3812

…ject reference

Fixes an issue where DatasetMemoryEntry.update() stored the direct
object reference (this.data = data) rather than a JSON snapshot.
When the same mutable object was pushed multiple times with mutations
between pushes, all stored entries reflected the final mutated state.

This is inconsistent with the file-system backed implementation
(DatasetFileSystemEntry), which serializes to JSON on write and
parses on read — naturally creating independent copies.

Changed this.data = data to this.data = JSON.parse(JSON.stringify(data)).
Added test verifying independent snapshots for mutable objects.

Co-authored-by: atlarix-agent <agent@atlarix.dev>
@AmariahAK

This comment was marked as spam.

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your interest in this project @AmariahAK !

The changes you are proposing are noop - by the time the data reaches the DatasetMemoryEntry.update method, it's been already serialized by the Dataset.pushData method ⬇️

const payload = checkAndSerialize(data, limit);

See that the original issue is for the v4 branch, where the storage management has been revamped. The issue doesn't present in the current master (v3).

Closing this PR, feel free to open another one against the v4 branch, if you feel like it.

@barjin barjin closed this Jul 7, 2026
@AmariahAK

Copy link
Copy Markdown
Contributor Author

@barjin , i made a fix to this pr #3831

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v4: MemoryStorageClient's DatasetClient stores records by reference

5 participants