refactor: move lambda variable scope into Physical Planning Context

[sweb]

Which issue does this PR close?

• Closes Move lambda evaluation into Physical Planning Context #23700 .

Rationale for this change

This is a clean up following #23649

What changes are included in this PR?

• Moves the lambda-variable scope state out of ExecutionProps and into PhysicalPlanningContext, following the same pattern as the scalar-subquery state in refactor: pass PhysicalPlanningContext explicitly through planner traits #23649.

• Puts PhysicalPlanningContext::indexes behind an Arc so the new per-lambda-body clone doesn't deep-copy the subquery index map. PhysicalPlanningContext::new's signature is unchanged.

Are these changes tested?

• New lambda_variables_shadow_outer_scope unit test in physical_planning_context.rs covers with_qualified_lambda_variables / lambda_variable_qualifier directly, including an inner lambda shadowing an outer parameter name.

Are there any user-facing changes?

this is a breaking change for lib users.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v54.1.0 (current)
       Built [  31.364s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.074s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  26.749s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.075s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.322s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/inherent_method_missing.ron

Failed in:
  ExecutionProps::with_qualified_lambda_variables, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/d3ec9a33c40687a3c549658a2660d85530540d4a/datafusion/expr/src/execution_props.rs:145

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field lambda_variable_qualifier of struct ExecutionProps, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/d3ec9a33c40687a3c549658a2660d85530540d4a/datafusion/expr/src/execution_props.rs:66

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  60.790s] datafusion-expr
    Building datafusion-physical-expr v54.1.0 (current)
       Built [  29.016s] (current)
     Parsing datafusion-physical-expr v54.1.0 (current)
      Parsed [   0.048s] (current)
    Building datafusion-physical-expr v54.1.0 (baseline)
       Built [  28.949s] (baseline)
     Parsing datafusion-physical-expr v54.1.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-physical-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.306s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  59.163s] datafusion-physical-expr

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.02%. Comparing base (f05869b) to head (6de3485).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23989      +/-   ##
==========================================
- Coverage   81.02%   81.02%   -0.01%     
==========================================
  Files        1106     1106              
  Lines      380692   380708      +16     
  Branches   380692   380708      +16     
==========================================
+ Hits       308468   308470       +2     
- Misses      53997    54008      +11     
- Partials    18227    18230       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sweb]

@timsaucer may I ask you for a review? Thank you!

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Refactors physical planning to move lambda-variable qualifier state out of ExecutionProps into PhysicalPlanningContext, aligning it with prior scalar-subquery state refactors.

Changes:

• Move lambda-variable qualifier state from ExecutionProps to PhysicalPlanningContext.
• Put PhysicalPlanningContext::indexes behind Arc to avoid deep copies when cloning per lambda body.
• Update docs and add a unit test for lambda variable shadowing across nested scopes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
docs/source/library-user-guide/upgrading/55.0.0.md	Documents the breaking API move from ExecutionProps to PhysicalPlanningContext (now including lambda-variable qualifier state).
datafusion/physical-expr/src/planner.rs	Switches lambda-variable qualifier lookup/propagation to use PhysicalPlanningContext.
datafusion/expr/src/physical_planning_context.rs	Adds lambda-variable qualifier tracking to the planning context; wraps subquery indexes in Arc; adds unit test.
datafusion/expr/src/execution_props.rs	Removes lambda-variable qualifier state and helper API from ExecutionProps; updates Debug test.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sweb]

addressed in 678073f

[sweb]

addressed in 678073f

[sweb]

Not sure this concern holds up - this is planning time and once per lambda - not per row. Also, I would not expect massive parameter lists of lambdas or lambda chains - from my point of view due to the Arc this has already reduced clone cost.

[gstvg]

I agree with your points, this is minor, and the existing code already performs the full clone anyway. It also would require changing the received planning_ctx to a mutable &mut PhysicalPlanningContext

[timsaucer]

Looks good. @gstvg would you mind taking a look as the original author of the lambda work?

[gstvg]

LGTM thanks @sweb for your work and @timsaucer for the ping