perf: null-free fast path for COUNT(DISTINCT) primitive accumulator - #23956
Merged
Conversation
`PrimitiveDistinctCountAccumulator::update_batch` iterated values with a per-element `Option`/validity check even when the input array had no nulls. Add a fast path that, when `null_count() == 0`, inserts directly from the values buffer (mirroring what `merge_batch` already does), skipping the per-element branch. This is the same style of null-free fast path recently applied to the median / percentile_cont accumulators (apache#23954). Benchmarks (`count_distinct`, null-free 8192-row batches): count_distinct i64 80% distinct -66.4% (54.2 -> 18.4 us) count_distinct i64 99% distinct -66.3% (54.7 -> 18.4 us) count_distinct u32 80% distinct -68.7% count_distinct u32 99% distinct -62.9% count_distinct i32 80% distinct -64.4% count_distinct i32 99% distinct -63.7% Co-authored-by: Claude Code
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23956 +/- ##
==========================================
- Coverage 80.69% 80.69% -0.01%
==========================================
Files 1095 1095
Lines 372529 372556 +27
Branches 372529 372556 +27
==========================================
+ Hits 300626 300643 +17
- Misses 53942 53946 +4
- Partials 17961 17967 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
neilconway
approved these changes
Jul 29, 2026
neilconway
left a comment
Contributor
There was a problem hiding this comment.
lgtm! Nice perf win for such a narrowly scoped change.
Member
Author
|
Thanks @neilconway for review |
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
…pache#23956) ## Which issue does this PR close? - Closes apache#23955. ## Rationale for this change `PrimitiveDistinctCountAccumulator::update_batch` does a per-element validity check on every row even when the array has no nulls. Its own `merge_batch` already inserts wholesale from the values buffer, so the same fast path applies to `update_batch`. Inspired by the null-free fast paths added to `percentile_cont`/`median` in apache#23954. ## What changes are included in this PR? - When `null_count() == 0`, insert directly from the values buffer (`self.values.extend(arr.values().iter().copied())`) instead of the per-element `Option` check. - Unit test asserting the fast and general paths agree (a dense array and a null-containing array with the same non-null values yield the same distinct count). ## Benchmarks `count_distinct` benchmark, null-free 8192-row batches: ``` count_distinct i64 80% distinct -67.0% (54.9 -> 18.1 us) count_distinct i64 99% distinct -66.0% (54.9 -> 18.7 us) count_distinct u32 80% distinct -66.9% (53.9 -> 17.8 us) count_distinct u32 99% distinct -65.3% (53.3 -> 18.5 us) count_distinct i32 80% distinct -66.9% (54.1 -> 17.9 us) count_distinct i32 99% distinct -66.2% (53.2 -> 18.0 us) ``` The bitmap-backed cases (u8/i8/u16/i16) and the grouped-accumulator cases don't use this path and are unchanged, as expected. ## Are these changes tested? Yes — new unit test; existing count-distinct tests pass. ## Are there any user-facing changes? No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
PrimitiveDistinctCountAccumulator::update_batchdoes a per-element validity check on every row even when the array has no nulls. Its ownmerge_batchalready inserts wholesale from the values buffer, so the same fast path applies toupdate_batch. Inspired by the null-free fast paths added topercentile_cont/medianin #23954.What changes are included in this PR?
null_count() == 0, insert directly from the values buffer (self.values.extend(arr.values().iter().copied())) instead of the per-elementOptioncheck.Benchmarks
count_distinctbenchmark, null-free 8192-row batches:The bitmap-backed cases (u8/i8/u16/i16) and the grouped-accumulator cases don't use this path and are unchanged, as expected.
Are these changes tested?
Yes — new unit test; existing count-distinct tests pass.
Are there any user-facing changes?
No.