Repository navigation
Update ClusterByStatisticsCollectorImpl to use bytes instead of keys - #12998
Merged
abhishekagarwal87 merged 12 commits intoOct 3, 2022
Merged
abhishekagarwal87 merged 12 commits into
abhishekagarwal87 merged 12 commits into
Conversation
cryptoe
reviewed
Sep 12, 2022
cryptoe
approved these changes
Sep 15, 2022
cryptoe
left a comment
cryptoe
left a comment
Contributor
There was a problem hiding this comment.
Minor comments.
Will approve once they are addressed.
Thanks for this patch @adarshsanjeev. This would be super usefull.
cryptoe
reviewed
Sep 15, 2022
cryptoe
left a comment
cryptoe
left a comment
Contributor
There was a problem hiding this comment.
Reverting the accidental approve :)
cryptoe
approved these changes
Sep 19, 2022
abhishekagarwal87
approved these changes
Oct 3, 2022
This was referenced Dec 18, 2022
riovic918data
pushed a commit
to riovic918data/druid
that referenced
this pull request
Jun 12, 2026
…pache#12998) * Update clusterByStatistics to use bytes instead of keys * Address review comments * Resolve checkstyle * Increase test coverage * Update test * Update thresholds * Update retained keys function * Update docs * Fix spelling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, ClusterByStatisticsCollectorImpl uses number of keys retained as a metric to check when to downsample a bucket. The size of keys is not necessarily the best metric, if the size of keys is small, more than the threshold could fit in memory. If the size is very large, it could lead to out of memory issues. This could be improved by changing this to check for the number of bytes currently retained instead. This would let us avoid downsampling if the keys are small, which would result in better accuracy for key collector.
The K value (which determines accuracy) for Quantile sketches which are used by KeyCollectors is set at 4096, which potentially could be higher, since we can downsample (which reduces the K). This has also been increased to 32768 for greater accuracy (at the cost of more memory).
Changes
Memory Usage
Currently, the threshold for keys per ClusterByStatisticsCollector is 65,536. Once all sketches have a combined total of this threshold, sketches are downsampled till the total keys retained is halved. The new byte threshold for ClusterByStatisticsCollector is 300MB. For keys of average length 300 (which is around one time column and 8 string columns of 30 characters each), we would have 300MB/300 or 1,000,000 keys.
For quantiles sketches, the K has been increased to 32768, which comes around to 11 * 32,768 or 360,448 keys. Considering a key as 300 bytes again, this comes out to 108MB.
For DistinctKeyCollectors, the initial byte threshold before downsampling is 100MB.
Key changed/added classes in this PR
ClusterByStatisticsCollectorImplDistinctKeyCollectorQuantilesSketchKeyCollectorDelegateOrMinKeyCollectorThis PR has: