Repository navigation
High buffer Queue causing OutOfMemory Exception #716
Description
In Our project we have a scenario where we are polling parallel consumer in batch size of 200 and from the batch we are performing some filtering and processing the records. So our lambda for the poll is not very fast(has some map creation based on topic segregation and an rest call for each topic).
With concurrency of 2 and batch size of 200 we are getting OutOfMemory exception with more than 3mil consumer record objects in the heap.
As per our observation we dont see the loadfactor stepdown scenario and seems like parallel consumer is loading the memory faster than we are consuming. We want to have an configuration for memory upper limit. to avoid OutOfMemory issue due to overfetch.
Activity
rkolesnev commentedon Feb 15, 2024
Hi PRATIK SATAPATHY (@pratiksatapathy),
Can you confirm what is the version of Parallel Consumer that you are using?
There was a bug in calculating queue size based on batch size - which would effectively use batchSize^2 * maxConcurrency * load factor - that was fixed in #662.
I think you probably still have the version without the fix above - if you are seeing over 3M objects in heap with batch size 200 and 2 concurrency. As with even full 100 load factor you should have a ceiling of 40K (200 * 100 * 2) instead.
In addition in last release i have just added couple of new options to set buffer size explicitly or override both min and max load factors - #682.
The load factor doesn't have a stepdown - it is a self tuning optimisation to figure the buffer size under peak load.
The logic is more along the lines of: start from lower value - step up until buffer size is sufficient under peak load - stay at that value going forward - as we cannot anticipate the spikes in load - so buffer size is kept to accommodate max previously seen spike.
I would suggest to try it first without changing load factor first - just with the #662 fix in place. And only if you need to reduce the buffer size further - then use options to change min / max load factor added in #682.
Hope that helps.
rkolesnev commentedon May 23, 2024
stale. closing.