Repository navigation
Memory accounting inaccuracies in multi level merge #23760
Description
Describe the bug
Not enough memory is reserved for the SortPreservingMerge
- track the record batch buffering (even with buffer_size=1 there's one record batch being prefetched by spawn_buffer)
- take into account that for the minimum requirement for two stream merge, we need to reserve 3 record batches because there could be a record batch crossing happening inside one of the streams
- for multi-column sort, take into account the additional overhead of ReusableRows
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
ariel-miculas commentedon Jul 21, 2026
take
ariel-miculas commentedon Jul 22, 2026
There's also an inconsistency between the pre-reservation done in multi level merge and the reservations in SortPreservingMerge that end up being ignored because a bypass memory pool is used.
ariel-miculas commentedon Jul 22, 2026
Also borrow this idea from #14748 :
except we don't have to do the row conversion since it should have been done in a previous step, figure out how to reuse it.
ariel-miculas commentedon Aug 25, 2026
this was addressed by #23802actually this only reduced the extra
Rows, but the memory for the RowConverter is still not taken into account