Repository navigation
Conversation
…ject reference Fixes an issue where DatasetMemoryEntry.update() stored the direct object reference (this.data = data) rather than a JSON snapshot. When the same mutable object was pushed multiple times with mutations between pushes, all stored entries reflected the final mutated state. This is inconsistent with the file-system backed implementation (DatasetFileSystemEntry), which serializes to JSON on write and parses on read — naturally creating independent copies. Changed this.data = data to this.data = JSON.parse(JSON.stringify(data)). Added test verifying independent snapshots for mutable objects. Co-authored-by: atlarix-agent <agent@atlarix.dev>
This comment was marked as spam.
This comment was marked as spam.
barjin
reviewed
Jul 7, 2026
barjin
left a comment
Member
There was a problem hiding this comment.
Thank you for your interest in this project @AmariahAK !
The changes you are proposing are noop - by the time the data reaches the DatasetMemoryEntry.update method, it's been already serialized by the Dataset.pushData method ⬇️
crawlee/packages/core/src/storages/dataset.ts
Line 285 in 15f9e55
See that the original issue is for the v4 branch, where the storage management has been revamped. The issue doesn't present in the current master (v3).
Closing this PR, feel free to open another one against the v4 branch, if you feel like it.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DatasetMemoryEntry.update()stores a JSON snapshot of the provideddata instead of the direct object reference. Previously, when the same
mutable object was pushed multiple times with mutations between pushes,
all stored entries reflected the final mutated state because they
shared the same reference.
This is inconsistent with the file-system backed implementation
(
DatasetFileSystemEntry), which serializes to JSON on write andparses on read — naturally creating independent copies.
The fix changes
this.data = datatothis.data = JSON.parse(JSON.stringify(data))inpackages/memory-storage/src/fs/dataset/memory.ts.A test (
dataset-snapshot.test.ts) verifies that each push creates anindependent snapshot by pushing a mutable object three times with
mutations between pushes and asserting each stored entry preserves its
push-time value.
Related: #3809 (stale references held in memory may prevent GC — the
snapshot approach helps by breaking reference chains, though that issue
is not fully resolved by this change alone).
Closes #3812