Enable dynamic filters for range-partitioned joins - #23854
Conversation
…/hash-join-dynamic-filter-with-range-partition
|
cc @gene-bordegaray @jayshrivastava @stuhood @adriangb — kindly pinging on this one. Please take a look if you have time, thanks! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23854 +/- ##
==========================================
+ Coverage 81.05% 81.06% +0.01%
==========================================
Files 1106 1107 +1
Lines 382287 382052 -235
Branches 382287 382052 -235
==========================================
- Hits 309851 309710 -141
+ Misses 54121 54058 -63
+ Partials 18315 18284 -31 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
@comphead We’d need to enable range partitioning on the join inputs for the effect to show up. |
getChan
left a comment
getChan
left a comment
There was a problem hiding this comment.
make sense to me!
If it's not too much trouble, could you share some performance comparison benchmark results?
this still need some additional plumbing to really work, but let me try to do that and run some benchmark |
|
I believe there is much discussion going on around the future of dynamic filters here #23814 . I am unable to give PRs full attention this week but think syncing with @jayshrivastava will be very useful |
…mic-filter-with-range-partition
9056b30 to
bf4e6a1
Compare
|
Hi @peterxcli, thanks for the request (#23854 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: 2010YOUY01, Dandandan, Fokko, Jefffrey, Omega359, Rachelint, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas. File an issue against this benchmark runner |
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
I looked at #23814, IIUC, it is orthogonal to this patch, it only discussed how to define the api so other nodes can receive the probe side fragment plan and dynamic filter of hash join, because what we're doing here is still building |
Currently the tpch also I'm looking into #23093. |
|
@peterxcli It doesn't have to be TPC-H—we could also add some focused benchmarks under datafusion/physical-plan/benches. |
jayshrivastava
left a comment
jayshrivastava
left a comment
There was a problem hiding this comment.
These code changes look good! Sorry for the delay in reviewing.
I think we could use 3 more tests. I left comments about 2 tests below - pretty small changes.
It would be nice to have an e2e test showing that this works. There's a test here:
which you can probably copy for range partitioning.| { | ||
| Arc::clone(&real_branches[0].1) | ||
| Arc::clone(&partition_filters[real_partition_ids[0]]) | ||
| } else if let Some(range_partitioning) = &self.probe_range_partitioning { |
There was a problem hiding this comment.
Nice this is all much easier to read.
jayshrivastava
left a comment
jayshrivastava
left a comment
There was a problem hiding this comment.
Nice. Left a nit, but this LGTM!
- You might want to bump this in discord or at the community meeting tomorrow so a maintainer can take a look.
- I think a sqllogictest should be possible and would be nice, but it's not blocking because the existing test is good.
- In a separate issue, it would be nice to add benchmarks for range partitioning. I don't think there's any. Can you file one under #22395?
| // Not portable to sqllogictest: this test pins `PartitionMode::Partitioned` | ||
| // by hand-wiring matching Range repartitioning on both join sides, which the | ||
| // SQL planner does not currently produce. | ||
| #[tokio::test] |
There was a problem hiding this comment.
I don't think you need this comment. It's totally fine to construct the plan manually. The Hash partitioned test in this file does the same.
For the record, I think there should be a way to get range partitioning working in sqllogictest? See here:
There was a problem hiding this comment.
Thanks for the pointer, but seems like the slt is using csv as data source, which cant do filter pushdown.
There was a problem hiding this comment.
I see, ya it is using a csv backed table but we could either:
- update it to be parquet backed and reuse them for your testing (I would prefer this is the code changes are small)
- or could introduce a
ParquetSourcein the PR and add coverage there
For range partitioning I would really prefer there to be coverage for all features in this file 🙇 .
NOTE: I would still keep the existing tests you have, they are great
| Some(9), | ||
| None, | ||
| None, | ||
| ])), |
|
@jayshrivastava thanks for the great review! |
gene-bordegaray
left a comment
gene-bordegaray
left a comment
There was a problem hiding this comment.
Thank you for this work, exciting to see this being pushed forward 🙇
|
|
||
| #[tokio::test] | ||
| async fn test_hashjoin_dynamic_filter_pushdown_range_partitioned() { | ||
| use datafusion_common::JoinType; |
There was a problem hiding this comment.
nit: could we keep these at the module level?
There was a problem hiding this comment.
I followed how test_hashjoin_hash_table_pushdown_partitioned's pattern
datafusion/datafusion/core/tests/physical_optimizer/filter_pushdown.rs
Lines 948 to 949 in 18f9db3
it also import the similar things at the top of test function.
should I also move their use to module level, or just follow their convention?
There was a problem hiding this comment.
since it seems we are repeating imports, would probably be good to add these to module level, thaknk you 🙇
| // https://github.com/apache/datafusion/issues/20195 | ||
| if config.optimizer.preserve_file_partitions > 0 | ||
| && self.mode == PartitionMode::Partitioned | ||
| && !matches!( |
There was a problem hiding this comment.
I think this might read easier if we chcked if it is Partitioning::Hash instead, what do you think?
Could we also update the comment explaining what we allow and what we do not as of now and the reasons why (partition / filter alignment)
| // Not portable to sqllogictest: this test pins `PartitionMode::Partitioned` | ||
| // by hand-wiring matching Range repartitioning on both join sides, which the | ||
| // SQL planner does not currently produce. | ||
| #[tokio::test] |
There was a problem hiding this comment.
I see, ya it is using a csv backed table but we could either:
- update it to be parquet backed and reuse them for your testing (I would prefer this is the code changes are small)
- or could introduce a
ParquetSourcein the PR and add coverage there
For range partitioning I would really prefer there to be coverage for all features in this file 🙇 .
NOTE: I would still keep the existing tests you have, they are great
| .iter() | ||
| .zip(partition_filters) | ||
| .map(|(split_point, then_expr)| { | ||
| let when_expr = build_lexicographic_filter( |
There was a problem hiding this comment.
we need to be careful here. I don't think that this is always producing the same ordering as RangePartitioning. Codex caught a nasty edge case.
The repartitioned for range uses compare_rows which uses ScalarValue::try_cmp, build_lexicographic_filter uses a binary expression comparisions which do not have 1-1 semantics.
An edge case is comparing floating point 0.0 values. If a split point was 0.0 then the bounds of range partitioning would be:
partition 0: key < +0.0
partition 1: +0.0 <= key
ScalarValue::try_cmp will evaluate -0.0 to route to partition 0 since it thinks of -0.0 < +0.0 as true. The binary expression comparision will normalize these values to 0.0 and have -0.0 < +0.0 as false, thus would evaluate against the wrong filter.
Can we create a 1-1 mirroring of the compare_rows sematntics and add a regression test here.
Here is a pointer to the compare_rows I am talking about: https://github.com/peterxcli/datafusion/blob/18f9db399331ef1c2163c8d5a230b7169d073e50/datafusion/common/src/utils/mod.rs#L120
and here is where it is used for range repartitioner: https://github.com/peterxcli/datafusion/blob/18f9db399331ef1c2163c8d5a230b7169d073e50/datafusion/physical-plan/src/repartition/mod.rs#L962
There was a problem hiding this comment.
IMO, because now we're using CASE WHEN expression to do the partition routing, so I'm not sure that 1:1 replicate from
datafusion/datafusion/common/src/utils/mod.rs
Lines 119 to 146 in 18f9db3
Maybe we should just reject this case in
There was a problem hiding this comment.
Well I think we can achieve this if we redesign the statement a bit.
Right now we do:
CASE
WHEN key <range split[0] THEN F0
WHEN key <range split[1] THEN F1
...
ELSE Fn
ENDand the floating point comparisns here are cuasing us to suffer.
Range partitioning guarentees that a row will be routed to a particular partition number based on the split points and the row values itself. So we can modify this statement to reflect that rather that comparing floating points directly in the case, with someting like this:
CASE range_partition(key)
WHEN 0 THEN F0
WHEN 1 THEN F1
...
ELSE Fn
ENDWe can extract the code that determines where a row will fall form the repartition code into a comon helper:
https://github.com/peterxcli/datafusion/blob/18f9db399331ef1c2163c8d5a230b7169d073e50/datafusion/physical-plan/src/repartition/mod.rs#L974-L992
Then we can use something similar to what the Hash case is doing with HashExpr here by adding a RangeExpr that will return the partition rows belong in. Then both the repartition and this code can call it to ensure behavior is the same.
I don't believe this will have any pruning regression from the prior form since the page-stats pruner doesnt understand CASE statements at all.
For CPU overhead we now have to extract a ScalarValue per key in every row but this was something we did in the repartitioner knowing we can improve in later iterations as benchmarks showed proof of needing better perf here. We can file an issue to benchmark this and samply it to find where / how we can improve. One idea is to turn the keys into Arrow arrays and do this once per batch.
@jayshrivastava @LiaCastaneda also lmk what you guys this
great suggestion. I will post this to discord after addressing Gene's review. |
|
@gene-bordegaray and @jayshrivastava -- do you think this PR is ready for review / merge? It is listed as something for the release in #22393 and I am trying to make sure everything is sorted |
I would wait until comments I have made are addressed. I can ping you. @peterxcli are you able to respond / address soon? |
|
Sure, I will address your review in few hours. |
|
Thanks -- I think by early next week also fine -- @timsaucer is hoping to make a RC mid next week |
There was a problem hiding this comment.
@gene-bordegaray thanks for the review.
There are two inline comments that I didn't resolve. I think we need more discussion around "edge case that is comparing floating point 0.0 values" especially.
|
|
||
| #[tokio::test] | ||
| async fn test_hashjoin_dynamic_filter_pushdown_range_partitioned() { | ||
| use datafusion_common::JoinType; |
There was a problem hiding this comment.
I followed how test_hashjoin_hash_table_pushdown_partitioned's pattern
datafusion/datafusion/core/tests/physical_optimizer/filter_pushdown.rs
Lines 948 to 949 in 18f9db3
it also import the similar things at the top of test function.
should I also move their use to module level, or just follow their convention?
| .iter() | ||
| .zip(partition_filters) | ||
| .map(|(split_point, then_expr)| { | ||
| let when_expr = build_lexicographic_filter( |
There was a problem hiding this comment.
IMO, because now we're using CASE WHEN expression to do the partition routing, so I'm not sure that 1:1 replicate from
datafusion/datafusion/common/src/utils/mod.rs
Lines 119 to 146 in 18f9db3
Maybe we should just reject this case in
|
|
||
| #[tokio::test] | ||
| async fn test_hashjoin_dynamic_filter_pushdown_range_partitioned() { | ||
| use datafusion_common::JoinType; |
There was a problem hiding this comment.
since it seems we are repeating imports, would probably be good to add these to module level, thaknk you 🙇
| ); | ||
| } | ||
|
|
||
| fn hashjoin_pushdown_scans() -> ( |
There was a problem hiding this comment.
really great thank you ❤️
| .iter() | ||
| .zip(partition_filters) | ||
| .map(|(split_point, then_expr)| { | ||
| let when_expr = build_lexicographic_filter( |
There was a problem hiding this comment.
Well I think we can achieve this if we redesign the statement a bit.
Right now we do:
CASE
WHEN key <range split[0] THEN F0
WHEN key <range split[1] THEN F1
...
ELSE Fn
ENDand the floating point comparisns here are cuasing us to suffer.
Range partitioning guarentees that a row will be routed to a particular partition number based on the split points and the row values itself. So we can modify this statement to reflect that rather that comparing floating points directly in the case, with someting like this:
CASE range_partition(key)
WHEN 0 THEN F0
WHEN 1 THEN F1
...
ELSE Fn
ENDWe can extract the code that determines where a row will fall form the repartition code into a comon helper:
https://github.com/peterxcli/datafusion/blob/18f9db399331ef1c2163c8d5a230b7169d073e50/datafusion/physical-plan/src/repartition/mod.rs#L974-L992
Then we can use something similar to what the Hash case is doing with HashExpr here by adding a RangeExpr that will return the partition rows belong in. Then both the repartition and this code can call it to ensure behavior is the same.
I don't believe this will have any pruning regression from the prior form since the page-stats pruner doesnt understand CASE statements at all.
For CPU overhead we now have to extract a ScalarValue per key in every row but this was something we did in the repartitioner knowing we can improve in later iterations as benchmarks showed proof of needing better perf here. We can file an issue to benchmark this and samply it to find where / how we can improve. One idea is to turn the keys into Arrow arrays and do this once per batch.
@jayshrivastava @LiaCastaneda also lmk what you guys this
10 participants
Which issue does this PR close?
Rationale for this change
Partitioned hash joins build one dynamic filter per build partition. Existing routing uses
hash(key) % N, which cannot reproduce a Range partitioning layout.Compatible Range co-partitioned joins instead need to route probe rows using their existing ordering and split points.
What changes are included in this PR?
CASEexpression that routes probe rows to the corresponding partition filter using the Range ordering and split points.Are these changes tested?
unit test.
Are there any user-facing changes?
no