You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Checking maxRowsPerSegment after segment merge in dynamic partitioning #11252
In this issue, the segment size refers to the row count in the segment. In dynamic partitioning, the task compares maxRowsPerSegment and the current Sink size to determine whether the segment hits the max segment size limit and needs to be pushed to deep storage. Here, the Sink size is the sum of in-memory segment size and the size of all persisted segments before they are merged. As a result, when the rollup is on, the Sink size can be larger than the actual segment size. This can create smaller segments than maxRowsPerSegment unnecessarily.
Activity