fix: Add Substrait roundtrip support for EXISTS and correlated OuterReferenceColumn - #18987
Closed
Nithurshen wants to merge 5 commits into
Closed
fix: Add Substrait roundtrip support for EXISTS and correlated OuterReferenceColumn#18987Nithurshen wants to merge 5 commits into
Nithurshen wants to merge 5 commits into
Conversation
Contributor
|
I'm requesting some backup for reviewing this one, thanks for the contribution! |
vbarua
reviewed
Jan 9, 2026
vbarua
left a comment
vbarua
left a comment
Contributor
There was a problem hiding this comment.
I don't think this is the best approach to fix this, primarily because you've added a non-standard mechanism to handle outer references. The plans you produce will only work on DataFusion, and not other systems.
If you want to see what this would look like in more standard Substrait, you can use the isthmus tool with the following input:
isthmus \
--create "CREATE TABLE lsaj_t1(t1_id INT, t1_name VARCHAR, t1_int INT)" \
--create "CREATE TABLE lsaj_t2(t2_id INT, t2_name VARCHAR, t2_int INT)" \
"SELECT t1_id, t1_name
FROM lsaj_t1
WHERE NOT EXISTS (SELECT 1 FROM lsaj_t2 WHERE t1_id = t2_id)
ORDER BY t1_id"
to generate a Substrait plan that captures the outer reference using standard Substrait.
| }; | ||
|
|
||
| let fn_name = substrait_fun_name(fn_signature); | ||
| if fn_name == "outer_reference" { |
Contributor
There was a problem hiding this comment.
Substrait already has a mechanism for handling outer references via the OuterReference root_type inside of a FieldReference.
Defining a custom function like you have here means that other systems won't be able to understand these plans, and DataFusion won't understands plans from other systems that use the standard mechanism for outer references.
Smallfu666
pushed a commit
to Smallfu666/datafusion
that referenced
this pull request
Aug 18, 2026
…field references (apache#23488) ## Which issue does this PR close? - Closes apache#16280. ## Rationale for this change The Substrait producer errors on `Expr::OuterReferenceColumn`, so any plan containing a correlated subquery cannot be serialized currently. DataFusion's round-trip tests don't hit this because the optimizer decorrelates subqueries into joins before serialization, but any workflow that serializes *unoptimized* plans (e.g. sending raw plans between systems for later optimization in my case) fails on queries like TPC-H q2/q4/q17/q20/q21/q22, and ~26 cases in `joins.slt` fail in `--substrait-round-trip` mode. A previous attempt (apache#18987) was closed because it introduced a non-standard mechanism for outer references, producing plans only DataFusion would be able consume but Substrait already represents correlated references natively via a `FieldReference` with an `OuterReference` root type and a `steps_out` depth. The consumer side of was implemented in apache#20439, which resolves `OuterReference` field references against a stack of outer schemas. This PR implements the producing half, symmetric with that design, so correlated plans round-trip using only standard Substrait. ## What changes are included in this PR? - `SubstraitProducer` gains outer-schema-stack methods mirroring `SubstraitConsumer`: `push_outer_schema` / `pop_outer_schema` (default no-ops) and `get_outer_schema(steps_out)` (default `None`), plus a `handle_outer_reference_column` method so custom producers can override the behaviour like every other expression kind. Defaults are backward compatible: existing custom producers are unaffected unless a plan actually contains an outer reference, in which case they now get an actionable error instead of `not_impl_err`. - `DefaultSubstraitProducer` maintains the stack in a `Vec<DFSchemaRef>`. - The four subquery producers (`from_in_subquery`, `from_scalar_subquery`, `from_exists`, `from_set_comparison`) push the enclosing query's schema around the subquery plan conversion (via a shared `produce_subquery_rel`, analogous to the consumer's `consume_subquery_rel`). - `from_outer_reference_column` (previously unused and emitting an incorrect plain `RootReference`) now resolves the column against the outer-schema stack, innermost first, and emits a `FieldReference` with an `OuterReference` root and the corresponding `steps_out`. - `to_substrait_rex` dispatches `Expr::OuterReferenceColumn` to the new handler instead of erroring. ## Are these changes tested? Yes: - New round-trip tests in `roundtrip_logical_plan.rs` covering correlated `EXISTS`, correlated `IN` subquery, correlated scalar subquery, and a nested correlated subquery that crosses two subquery boundaries (`steps_out = 2`). Each test asserts the produced plan contains an `OuterReference` at the expected depth and that the plan round-trips through the existing consumer with its schema intact. - Consumer-side resolution was already covered by the tests added in apache#20439; these tests now exercise both halves together. - `joins.slt` in `--substrait-round-trip` mode goes from 38 failures to 12; the remaining failures are pre-existing gaps unrelated to outer references (`USING` join constraint, plan-level lateral `LogicalPlan::Subquery`, duplicate unqualified field names). ## Are there any user-facing changes? - Plans containing correlated subqueries now serialize instead of returning "not implemented", emitting spec-standard `OuterReference` field references. - `SubstraitProducer` has three new provided methods and `handle_outer_reference_column`; all have defaults, so existing implementations continue to compile. - The signature of the public helper `from_outer_reference_column` changed (it now takes the producer and the outer field) — its previous form resolved against the wrong schema and emitted a plain `RootReference`, and it was not called from anywhere in the crate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Substrait roundtrip tests were failing for queries involving
EXISTSand correlated subqueries (specifically those using outer references). The DataFusion Substrait producer did not support serializingExpr::Exists, and it threw a "feature not implemented" error forOuterReferenceColumn.Supporting these features is essential for improving the reliability of DataFusion's Substrait integration and enabling complex join scenarios in distributed environments.
What changes are included in this PR?
Producer Support for
EXISTS:from_existsin the producer to mapExpr::Existsto the SubstraitSetPredicate(usingPredicateOp::Existence).NOT EXISTSby wrapping the predicate in a "not" scalar function.Producer Support for
OuterReferenceColumn:OuterReferenceColumnas a custom scalar function named"outer_reference". This avoids schema validation errors during serialization since outer columns often cannot be resolved against the local subquery schema.Consumer Support for
OuterReferenceColumn:"outer_reference"scalar function and deserialize it back into a DataFusionOuterReferenceColumn, ensuring a successful roundtrip.Are these changes tested?
Yes.
joins.slt.cargo test --test sqllogictests -- --substrait-round-trip joins.slt:1233Are there any user-facing changes?
No breaking API changes. This PR purely expands the coverage of supported Logical Plans that can be converted to/from Substrait.