Skip to content

Checking maxRowsPerSegment after segment merge in dynamic partitioning #11252

Open

Description

@jihoonson

Description

In this issue, the segment size refers to the row count in the segment. In dynamic partitioning, the task compares maxRowsPerSegment and the current Sink size to determine whether the segment hits the max segment size limit and needs to be pushed to deep storage. Here, the Sink size is the sum of in-memory segment size and the size of all persisted segments before they are merged. As a result, when the rollup is on, the Sink size can be larger than the actual segment size. This can create smaller segments than maxRowsPerSegment unnecessarily.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions