Doesn't support 1GB+ heapdumps #77

Activity

1. JacksonGL commented on May 19, 2023

   

   JacksonGL

   on May 19, 2023

   Member

   More actions

   @FexileTV Would it be possible for you to share the 1.9GB snapshot? Or, as an alternative, could you get a smaller snapshot that still reproduces the problem?
2. fx1738 commented on May 19, 2023

   

   fx1738

   on May 19, 2023

   Author

   More actions

   @FexileTV Would it be possible for you to share the 1.9GB snapshot? Or, as an alternative, could you get a smaller snapshot that still reproduces the problem?

   The snapshot was taken from an production environment so it contains sensitive information which I'm not able to share.

   The app uses .5GB under normal load (no memory leak) and when the rare leak does occur it instantly jumps to 3x the memory usage. So I'm unable to capture a smaller size heapsnapshot
3. JacksonGL commented on May 19, 2023

   

   JacksonGL

   on May 19, 2023

   Member

   More actions

   Without a repro of the issue, it will be challenging to identify the root cause and to validate the fix. The execution appears to stall at the dominator tree construction stage (it's not a crash, otherwise the JS runtime should exit).

   I may implement a CLI option to bypass this particular step later. This could allow MemLab to work without the retained size data for each object within the heap. Consequently, it won't rank the memory leaks based on their potential size.
4. fx1738 commented on May 20, 2023

   

   fx1738

   on May 20, 2023

   Author

   More actions

   I believe the issue could be reproduced by using any large heap snapshot.

   The option to bypass the dominator tree might work, however I'm unsure how far that would get me to seeing whats causing the problem within my app.

   I've managed to get a 1.3GB Heapsnapshot and let it run for 1 hour with the same issue:
   
5. JacksonGL commented on May 20, 2023

   

   JacksonGL

   on May 20, 2023

   Last edited by JacksonGL

   Member

   More actions

   The current JavaScript heap parser is not very good at processing very large heap snapshots, since memory leaks could often be captured by much smaller snapshots in most of use cases. If your JavaScript heap takes 500MB under normal load, a potential optimization is to encode the memory-consuming part into ArrayBuffer whenever possible. (ArrayBuffer is seen as external memory to the JavaScript heap, it will cause less pressure to garbage collector and significantly reduce the heap dump size).

   I guess your JavaScript heap is densely populated with objects, perhaps more so than usual. Consider trying the following steps to identify memory issues:

   1. Comment out the body of this method:
      

      memlab/packages/core/src/paths/TraceFinder.ts

      Line 590 in 0f0517e

      calculateAllNodesRetainedSizes(snapshot: IHeapSnapshot): void {

   2. Build memlab
   3. Run your local build of memlab with the following command to find objects with the highest number of outgoing edges:

   node packages/memlab/bin/memlab analyze object-fanout --snapshot <PATH/TO/YOUR/SNAPSHOT>

   4. View the retainer trace of those objects:

   node packages/memlab/bin/memlab view-heap --snapshot <PATH/TO/YOUR/SNAPSHOT> --node-id <HEAP-OBJECT-ID>

   Edit: I'm still having trouble replicating the freeze that occurs while handling big heap snapshots. Every time I try, Node.js just runs out of memory.