You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v4: MemoryStorageClient's DatasetClient stores records by reference #3812
letdata={rand: 0};asyncfunctionrefreshData(){data.rand=Math.random();}asyncfunctionpushAndRead({ storageClient }: {storageClient: StorageClient}){constdataset=awaitDataset.open(null,{ storageClient });refreshData();awaitdataset.pushData(data);refreshData();awaitdataset.pushData(data);refreshData();awaitdataset.pushData(data);console.log((awaitdataset.getData()).items);}awaitpushAndRead({storageClient: newMemoryStorageClient()});// memory storage client will print the same number three times, as all the items refer to the same object/// [/// { rand: 0.22326410406394825 },/// { rand: 0.22326410406394825 },/// { rand: 0.22326410406394825 }/// ]awaitpushAndRead({storageClient: newFileSystemStorageClient()});// file system client takes snapshots of the passed objects/// [/// { rand: 0.6922433682025506 },/// { rand: 0.33313558872226434 },/// { rand: 0.6864574308371908 }/// ]
DatasetClient implementations are inconsistent across different stock StorageClients.
MemoryStorageClient's Dataset will store the direct reference to the object instead of taking a JSON snapshot.
This can lead to unexpected mutations (see snippet above) or, e.g., unexpected GC consequences (see #3809).
All of the Crawlee issues are open for all contributors - feel free to send a PR!
Note that the problem described here is only present in the v4 branch - make sure to open the PR against this branch (not master) if you decide to work on this.
Activity
georgekimaninjoroge commentedon Jul 5, 2026
Is this open for external contributions? I'd be happy to take a look
barjin commentedon Jul 7, 2026
All of the Crawlee issues are open for all contributors - feel free to send a PR!
Note that the problem described here is only present in the
v4branch - make sure to open the PR against this branch (notmaster) if you decide to work on this.fix(memory-storage): store JSON snapshots in DatasetClient.pushData()
AmariahAK commentedon Jul 7, 2026
@barjin fixed it in this pr #3831
barjin commentedon Jul 10, 2026
Closed by #3831