Skip to content

SortMergeJoin projection pushdown can reuse stale JoinFilter indices #24414

Closed

Description

@haohuaijin

Describe the bug

SortMergeJoinExec may return incorrect results after projection pushdown when the join has a non-equi filter.

To Reproduce

set datafusion.optimizer.prefer_hash_join = false;
set datafusion.optimizer.repartition_joins = true;
set datafusion.execution.target_partitions = 2;

select t1.column2 as left_b1, t2.column2 as right_b1
from (values (100, 1, 0)) t1
join (values (10, 1)) t2
  on t1.column2 = t2.column2
 and t1.column1 > t2.column1;

The expected result is (1, 1), but the query returns no rows.

SortMergeJoinExec::try_swapping_with_projection projects the join inputs but reuses the original JoinFilter. Its column indices can therefore refer to the wrong columns in the new schemas.

The projection swap should rewrite the filter with update_join_filter, or be rejected when the filter columns are not projected.

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions