fix(proto): preserve HashJoinExec fetch across serialization by adriangb · Pull Request #24165 · apache/datafusion

adriangb · 2026-08-07T15:27:59Z

Which issue does this PR close?

Related to [EPIC] Port ExecutionPlan serialization to try_to_proto / try_from_proto hooks #23494 (found while auditing that EPIC's plans for unserialized
fields). This is a bug fix, not part of the migration checklist.

Rationale for this change

HashJoinExec.fetch was silently dropped by protobuf serialization.
protobuf::HashJoinExecNode had no fetch field, so HashJoinExec's
try_to_proto never wrote it and try_from_proto never restored it: a plan
with fetch = Some(n) round-tripped to fetch = None.

This is user-visible. The limit_pushdown physical optimizer rule pushes a
limit into the join via ExecutionPlan::with_fetch, then marks the global
state satisfied and drops the enclosing GlobalLimitExec. So after a proto
round-trip the plan carried no limit at all, and a distributed executor
(Ballista/Comet-style, anything that ships physical plans over the wire)
returned more rows than the query asked for.

What changes are included in this PR?

datafusion.proto: add optional uint64 fetch = 12 to HashJoinExecNode.

The field is presence-tracked on purpose, and this is the load-bearing
detail for wire compatibility. Messages written by versions predating this
field carry no fetch at all, and a plain proto3 scalar decodes that absence
as 0. With the negative-sentinel convention used by SortExecNode's
int64 fetch, 0 would mean "fetch 0 rows" and would silently turn every
older plan into an empty result. optional gives prost an Option<u64>
where absent decodes to None, which is the correct reading of an older
message. A comment in the .proto records this.
Regenerated prost.rs / pbjson.rs via datafusion/proto-models/regen.sh
(no hand edits).
hash_join/exec.rs: write self.fetch in the try_to_proto hook and
restore it in try_from_proto via the builder's with_fetch, matching how
the plan is normally constructed.
New regression test roundtrip_hash_join_fetch.

The deprecated PhysicalPlanNodeExt shims (try_from_hash_join_exec /
try_into_hash_join_physical_plan) delegate straight to these two hooks, so
they pick the fix up with no separate change. Verified by reading them rather
than assumed.

Are these changes tested?

Yes. roundtrip_hash_join_fetch in
datafusion/proto/tests/cases/roundtrip_physical_plan.rs builds a
HashJoinExec, applies with_fetch(Some(7)) the way limit_pushdown does,
round-trips it through physical_plan_to_bytes_with_proto_converter /
physical_plan_from_bytes_with_proto_converter, and asserts fetch() is
still Some(7). It also covers fetch = None.

The assertion deliberately inspects fetch() rather than the plan's string
form. The existing roundtrip_test helper compares format!("{plan:?}"), and
HashJoinExec's Debug output does not include fetch — which is exactly why
this went unnoticed. I confirmed this empirically: with the encode side
reverted, the Debug comparison inside the helper still passes and only the
fetch() assertion fails (left: None, right: Some(7)).

Ran locally:

cargo fmt --all
cargo test -p datafusion-proto --test proto_integration — 215 passed, 0 failed
cargo test -p datafusion-physical-plan — 1640 + 9 passed, 0 failed
cargo clippy --all-targets --all-features on the touched packages. The
changed code is clean; the only two errors reported are pre-existing on an
unmodified main with my newer local clippy (uninlined_format_args in
datafusion/proto-common/src/generated/pbjson.rs and
needless_pass_by_value in datafusion/proto/src/bytes/mod.rs), in files
this PR does not touch.

Are there any user-facing changes?

Yes, a bug fix: a limit pushed into a hash join now survives physical-plan
serialization, so distributed executors no longer over-return rows. No API
changes. The new proto field is backward and forward compatible in both
directions — old readers ignore tag 12, and new readers treat its absence as
"no limit".

`protobuf::HashJoinExecNode` had no `fetch` field, so `HashJoinExec`'s `try_to_proto` never wrote it and `try_from_proto` never restored it: a plan with `fetch = Some(n)` round-tripped to `fetch = None`. This is user-visible because the `limit_pushdown` physical optimizer rule pushes a limit into the join via `ExecutionPlan::with_fetch` and then drops the enclosing `GlobalLimitExec`. After a proto round-trip the plan therefore carried no limit at all, and a distributed executor returned more rows than the query asked for. Add `optional uint64 fetch = 12` and wire it through both hooks. The field is presence-tracked on purpose: messages written before it existed carry no `fetch`, and a plain proto3 scalar would decode that absence as `0` -- "fetch 0 rows" -- silently producing empty results. `optional` gives `None` for absent, which is the correct reading of an older message. The existing `roundtrip_test` helper cannot catch this class of bug: it compares `format!("{plan:?}")`, and `HashJoinExec`'s `Debug` output does not include `fetch`. The new regression test asserts on `fetch()` directly and covers both `Some(7)` and `None`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-08-07T15:37:25Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  44.135s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.132s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  35.675s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.132s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.869s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  83.411s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  56.903s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  57.105s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.327s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 116.274s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  23.888s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.123s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.056s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.126s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   2.335s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field HashJoinExecNode.fetch in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2040
  field HashJoinExecNode.fetch in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2040

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  52.042s] datafusion-proto-models

adriangb · 2026-08-07T15:50:43Z

Should we use a builder the whole way through instead of a try_new and then a builder later?

adriangb · 2026-08-07T15:51:05Z

+            hash_join = hash_join
+                .builder()


Let's avoid going from instance -> builder and back.

codecov-commenter · 2026-08-07T15:59:52Z

Codecov Report

❌ Patch coverage is 34.61538% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.05%. Comparing base (e2e8e1c) to head (a07b6a6).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	13 Missing ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	69.23%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24165    +/-   ##
========================================
  Coverage   81.05%   81.05%            
========================================
  Files        1107     1107            
  Lines      381407   381574   +167     
  Branches   381407   381574   +167     
========================================
+ Hits       309139   309275   +136     
- Misses      54013    54036    +23     
- Partials    18255    18263     +8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…roto Address review feedback: the decoder called `HashJoinExec::try_new` and then round-tripped through `.builder().with_fetch(..).build()` just to apply `fetch`. `try_new` is itself a thin wrapper over `HashJoinExecBuilder`, so construct through the builder directly and set `fetch` alongside the other options. No behavior change: `try_new` delegates to the same `HashJoinExecBuilder::new(..).with_filter(..).with_projection(..) .with_partition_mode(..).with_null_equality(..).with_null_aware(..) .build()` chain, so validation, `column_indices`, `join_schema` and the computed `PlanProperties` are identical. `with_dynamic_filter_expr` is a method on `HashJoinExec` (not the builder), so it remains a post-build step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb · 2026-08-07T16:24:15Z

@kosiew @kumarUjjawal would one of you be able to review this change please?

kumarUjjawal · 2026-08-07T16:51:02Z

+            // the join. The field is presence-tracked, so a message written
+            // before it existed decodes to `None` (no limit) rather than to
+            // `Some(0)`.
+            .with_fetch(hashjoin.fetch.map(|f| f as usize))


u64 as usize will silently truncates on 32-bit targets. A fetch of 1 << 32 will becomes 0.

kumarUjjawal · 2026-08-07T16:52:56Z

Thanks @adriangb

Left few comments

Review feedback: `u64 as usize` silently truncates on a 32-bit target, where `usize` is 32 bits. A fetch of `1 << 32` decodes to `0` -- not merely a wrong limit but the worst possible one, since "fetch 0 rows" turns the query into an empty result instead of erroring. Use `usize::try_from` and surface an out-of-range value as an error. Truncating and saturating both misrepresent the plan; an explicit decode failure is the honest outcome, and it is only reachable on a 32-bit target with an absurd fetch. `plan_datafusion_err!` rather than `internal_datafusion_err!`: this decode path reserves the internal-error macros for genuinely malformed nodes (an unknown `PartitionMode` discriminant, a dynamic filter that does not downcast), which really do indicate a bug. A well-formed but unrepresentable `fetch` is not a DataFusion bug -- it is a plan that cannot be expressed on this target -- and `plan_err!` is already this file's idiom for invalid plan configuration. The encode side (`self.fetch.map(|f| f as u64)`) needs no change: `usize` is at most 64 bits on every supported target, so widening to `u64` is lossless. `roundtrip_hash_join_fetch` now also covers `u32::MAX as usize` and `usize::MAX`, pinning that a large fetch round-trips exactly. Both are representable on every target, so the test stays portable. The truncation path itself is only reachable on a 32-bit target and is therefore not covered on a 64-bit CI host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch is stacked on the `fetch` fix and, after the rebase onto main, carries its own copy of the `HashJoinExec` decode path. Apply the same correction here so the defect does not survive on this branch and reappear once apache#24165 merges. Review feedback on apache#24165: `u64 as usize` silently truncates on a 32-bit target, where `usize` is 32 bits. A fetch of `1 << 32` decodes to `0` -- not merely a wrong limit but the worst possible one, since "fetch 0 rows" turns the query into an empty result instead of erroring. Use `usize::try_from` and surface an out-of-range value as an error. Truncating and saturating both misrepresent the plan; an explicit decode failure is the honest outcome, and it is only reachable on a 32-bit target with an absurd fetch. `plan_datafusion_err!` rather than `internal_datafusion_err!`: this decode path reserves the internal-error macros for genuinely malformed nodes (an unknown `PartitionMode` discriminant, a dynamic filter that does not downcast), which really do indicate a bug. A well-formed but unrepresentable `fetch` is not a DataFusion bug -- it is a plan that cannot be expressed on this target -- and `plan_err!` is already this file's idiom for invalid plan configuration. The encode side (`fetch.map(|f| f as u64)`) needs no change: `usize` is at most 64 bits on every supported target, so widening to `u64` is lossless. `roundtrip_hash_join_fetch` now also covers `u32::MAX as usize` and `usize::MAX`, pinning that a large fetch round-trips exactly. Both are representable on every target, so the test stays portable. The truncation path itself is only reachable on a 32-bit target and is therefore not covered on a 64-bit CI host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb · 2026-08-07T18:00:52Z

Thanks @kumarUjjawal, replied!

kumarUjjawal

Thanks for the response. Looks good 👍

github-actions Bot added proto Related to proto crate physical-plan Changes to the physical-plan crate labels Aug 7, 2026

github-actions Bot added the auto detected api change Auto detected API change label Aug 7, 2026

adriangb mentioned this pull request Aug 7, 2026

[EPIC] Port ExecutionPlan serialization to try_to_proto / try_from_proto hooks #23494

Open

24 tasks

adriangb commented Aug 7, 2026

View reviewed changes

adriangb mentioned this pull request Aug 7, 2026

refactor(proto): destructure plan and proto structs in join serde hooks #24164

Open

adriangb requested review from kosiew and kumarUjjawal August 7, 2026 16:23

kumarUjjawal reviewed Aug 7, 2026

View reviewed changes

kumarUjjawal approved these changes Aug 7, 2026

View reviewed changes

adriangb added this pull request to the merge queue Aug 7, 2026

Merged via the queue into apache:main with commit fc846dd Aug 7, 2026
40 checks passed

adriangb deleted the fix-proto-hash-join-fetch branch August 7, 2026 18:55

Navigation Menu

Uh oh!

fix(proto): preserve HashJoinExec fetch across serialization - #24165

fix(proto): preserve HashJoinExec fetch across serialization#24165
adriangb merged 3 commits into
apache:mainfrom
pydantic:fix-proto-hash-join-fetch

adriangb commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

adriangb Aug 7, 2026

Uh oh!

adriangb Aug 7, 2026

Uh oh!

codecov-commenter commented Aug 7, 2026 •

edited

Loading

Uh oh!

adriangb commented Aug 7, 2026

Uh oh!

kumarUjjawal Aug 7, 2026

Uh oh!

Uh oh!

Uh oh!

kumarUjjawal commented Aug 7, 2026

Uh oh!

adriangb commented Aug 7, 2026

Uh oh!

kumarUjjawal left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

adriangb commented Aug 7, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adriangb commented Aug 7, 2026

Uh oh!

kumarUjjawal Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kumarUjjawal commented Aug 7, 2026

Uh oh!

adriangb commented Aug 7, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Aug 7, 2026 •

edited

Loading

codecov-commenter commented Aug 7, 2026 •

edited

Loading