Physical-level placeholders #20009
Open
Physical-level placeholders #20009
+3,219
−44
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 26, 2026 11:57
Introduces `physical_expressions` and `with_physical_expressions` methods to the `ExecutionPlan` trait. These methods provide an interface to inspect and replace physical expressions within execution plan nodes. The commit also implements these methods for various ExecutionPlans and introduces a new physical expression invariant check to ensure consistency between expression retrieval and replacement.
This commit introduces physical-level support for placeholders, allowing physical plans to be prepared once and executed multiple times with different parameter values. Key changes: - Added `PlaceholderExpr` physical expression. - Added `ResolvePlaceholdersExec` to handle placeholder resolution at execution time. - Updated `TaskContext` to carry `ParamValues` for resolution during `execute`. - Added `PhysicalExprResolver` optimizer rule to manage `ResolvePlaceholdersExec` nodes. - Updated `FilterPushdown` to correctly handle expressions containing placeholders.
github-actions
bot
added
physical-expr
Author
|
@askalt, could you review the PR as well? |
Contributor
|
cc @Omega359 (I remember you were also interested in placeholders within physical plans). |
Contributor
Yes, I would like to be able to reuse plans - I'll try and find time in the next day or so to review this though I'm not particularly familiar with this area of code. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This PR covers a part of #14342.
Currently, DataFusion's support for placeholders is handled at the logical plan level. This requires re-planning every time parameter values change.
This PR introduces physical-level support for placeholders, allowing a physical plan to be created once and then executed multiple times with different parameter values. This should improve performance for repeated executions of the same query with different parameters by avoiding the overhead of physical planning.
What changes are included in this PR?
This PR introduces several key components to support physical placeholders:
physical_expressionsandwith_physical_expressionsto theExecutionPlantrait to allow inspection and replacement of expressions within plan nodes.PlaceholderExpr: A new physical expression that represents a placeholder in the physical plan. It stores the placeholder ID and its data type.ResolvePlaceholdersExec: A new execution plan node that acts as a wrapper. It identifies placeholders within its subtree and resolves them using parameter values provided at execution time.PhysicalExprResolverOptimizer Rule: Wraps the final optimized plan in aResolvePlaceholdersExecif any unresolved placeholders are detected.TaskContextUpdate: Updated to carryParamValues.Are these changes tested?
physical_expressionsandwith_physical_expressionsmethods are verified using thecheck_physical_expressionsinvariant.ResolvePlaceholdersExecis tested with unit tests and SLT tests.Are there any user-facing changes?
Yes, a user can now build physical plan from a logical plan with placeholders.