Repository navigation
Conversation
JIRA Issue Information=== Improvement SPARK-49110 === This comment was automatically generated by GitHub Actions |
cloud-fan
marked this pull request as draft
January 20, 2026 01:56
cloud-fan
changed the title
cloud-fan
marked this pull request as ready for review
January 20, 2026 05:59
cloud-fan
force-pushed
the
meta_col
branch
2 times, most recently
from
January 20, 2026 06:10
3ff52dd to
4921f9b
Compare
…opagate metadata columns ### What changes were proposed in this pull request? This PR simplifies `SubqueryAlias.metadataOutput` to always propagate metadata columns from its child, rather than only propagating when the child is a `LeafNode` or another `SubqueryAlias`. The previous implementation was introduced in SPARK-40149 as a workaround to forbid queries like `SELECT m FROM (SELECT a FROM t)` while still allowing DataFrame API chaining. However, this created an inconsistency since `SubqueryAlias` should conceptually just rename/qualify columns, not filter which ones are accessible. With this change: - `SubqueryAlias` always propagates `metadataOutput` (with qualifier applied) - The `qualifiedAccessOnly` filter is preserved to handle natural join metadata columns - Queries like `SELECT m FROM (SELECT a FROM t) AS alias` now work, consistent with how `Project` already propagates metadata columns ### Why are the changes needed? 1. **Consistency**: `SubqueryAlias` is a rename operation and should not selectively block metadata column propagation 2. **Simpler code**: Removes the special-case logic checking for `LeafNode`/`SubqueryAlias` children 3. **Better error messages**: When metadata columns from both sides of a join have the same name, users now get an "ambiguous reference" error rather than "column not found" ### Does this PR introduce _any_ user-facing change? Yes, queries that previously failed with "column not found" when accessing metadata columns through a subquery alias will now succeed (if unambiguous) or fail with "ambiguous reference" (if multiple columns have the same name). ### How was this patch tested? Updated existing tests and added new test for ambiguous metadata columns after join with SubqueryAlias.
cloud-fan
commented
Jan 20, 2026
allisonwang-db
approved these changes
Jan 20, 2026
Contributor
Author
|
thanks for the review, merging to master! |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR simplifies
SubqueryAlias.metadataOutputto always propagate metadata columns from its child, rather than only propagating when the child is aLeafNodeor anotherSubqueryAlias.The previous implementation was introduced in SPARK-40149 as a workaround to forbid queries like
SELECT m FROM (SELECT a FROM t)while still allowing DataFrame API chaining. However, this created an inconsistency sinceSubqueryAliasshould conceptually just rename/qualify columns, not filter which ones are accessible.With this change:
SubqueryAliasalways propagatesmetadataOutput(with qualifier applied)qualifiedAccessOnlyfilter is preserved to handle natural join metadata columnsSELECT m FROM (SELECT a FROM t) AS aliasnow work, consistent with howProjectalready propagates metadata columnsWhy are the changes needed?
SubqueryAliasis a rename operation and should not selectively block metadata column propagationLeafNode/SubqueryAliaschildrenDoes this PR introduce any user-facing change?
Yes, queries that previously failed with "column not found" when accessing metadata columns through a subquery alias will now succeed (if unambiguous) or fail with "ambiguous reference" (if multiple columns have the same name).
How was this patch tested?
Updated existing tests and added new test for ambiguous metadata columns after join with SubqueryAlias.
Was this patch authored or co-authored using generative AI tooling?
Yes.