Skip to content

[SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets - #46919

Closed
n-young-db wants to merge 2 commits into
apache:masterfrom
n-young-db:ineffective-rules-lazy
Closed

n-young-db wants to merge 2 commits into
apache:masterfrom
n-young-db:ineffective-rules-lazy

Conversation

@n-young-db

@n-young-db n-young-db commented Jun 7, 2024 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Changed the ineffectiveRules variable of the TreeNode class to initialize lazily. This will reduce unnecessary driver memory pressure.

Why are the changes needed?

  • Plans with large expression or operator trees are known to cause driver memory pressure; this is one step in alleviating that issue.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UT covers behavior. Outwards facing behavior does not change.

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions Bot added the SQL label Jun 7, 2024

@JoshRosen JoshRosen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

This is a nice optimization and the implementation is consistent with the pattern used for a similar optimization in #46285.

Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala Outdated
…s/TreeNode.scala

Co-authored-by: Josh Rosen <rosenville@gmail.com>
@JoshRosen JoshRosen closed this in 5a2f374 Jun 10, 2024
@JoshRosen

Copy link
Copy Markdown
Contributor

Merged to master (4.0.0). Thanks @n-young-db!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants