fix: support RunEndEncoded arrays in function and window frame type coercion by thorfour · Pull Request #24565 · apache/datafusion

thorfour · 2026-08-21T14:58:59Z

Which issue does this PR close?

Closes coerced_from and extract_window_frame_target_type doesn't support RunEndEncoded arrays #24564.

Rationale for this change

Adds missing support for coercing REE arrays and extracting range windows from REE arrays.

What changes are included in this PR?

Added case statements for supporting REE arrays

Are these changes tested?

Yes a unit test is included.

Are there any user-facing changes?

No

kumarUjjawal

Thank you @thorfour for working on this. I have left few comments please take a look.

kumarUjjawal · 2026-08-22T04:04:00Z

        Ok(DataType::Interval(IntervalUnit::MonthDayNano))
    } else if let DataType::Dictionary(_, value_type) = col_type {
        extract_window_frame_target_type(value_type)
+    } else if let DataType::RunEndEncoded(_, value_type) = col_type {


Unwrapping REE here also lets RANGE <offset> PRECEDING/FOLLOWING past planning. An offset frame adds the offset to each ordering value, and a value pulled from an REE column stays run-end wrapped, which arrow's arithmetic kernels don't handle. Keeping the plan time error until scalar arithmetic unwraps REE would be better.

Good call out, we could fix this in the window state code but agreed that it should get implemented in the arrow kernels

so whats the action that needs to be taken here

Some options I could think of:

We do nothing and wait for Arrow to implement this arithmetic.

We special case the windowing function to check if REE and perform the necessary arithmetic itself.

We add a step in planning to check if RANGE is being used on REE column and return the error in planning.

perhaps do no.3 then raise an upstream issue for no.1

thoughts @kumarUjjawal?

I've created the upstream issue: apache/arrow-rs#10891

Removed this branch, and added an slt test to ensure error at planning time happens for REE.

perhaps do no.3 then raise an upstream issue for no.1

thoughts @kumarUjjawal?

I was thinking we can put the unwrap back here, and make supports_offset_arithmetic false for a run-encoded order key? That keeps the plan time rejection for 1 PRECEDING and 1 FOLLOWING. An offset bound is the one case that needs add_checked on a wrapped value. The user then gets a plan error, not the internal "file a bug report" error.

Good idea! I've added that behavior.

Jefffrey

as mentioned would be good to have an end to end test via SLT

codecov-commenter · 2026-08-25T04:29:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.42%. Comparing base (ee59f62) to head (0d21a8d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #24565   +/-   ##
=======================================
  Coverage   81.42%   81.42%           
=======================================
  Files        1121     1121           
  Lines      402142   402178   +36     
  Branches   402142   402178   +36     
=======================================
+ Hits       327460   327493   +33     
- Misses      55484    55486    +2     
- Partials    19198    19199    +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

coerced_from() already special-cases Dictionary(_, value_type) so that functions declared with the old-style Exact/OneOf signature (which don't go through the newer Coercible/TypeSignatureClass system) match against the dictionary's value type. RunEndEncoded had no equivalent case, so any such function (e.g. date_bin) failed to plan when given a RunEndEncoded-wrapped column, even though the same column as a plain Dictionary would have matched fine. Add the symmetric RunEndEncoded case, unwrapping to the value field's type before recursing, mirroring the existing Dictionary handling.

extract_window_frame_target_type() already recurses through Dictionary(_, value_type) to find the natural type for a RANGE window frame bound, but had no equivalent case for RunEndEncoded, so ordering a RANGE window by a REE-dict-encoded column (the same column as a plain Dictionary would work) failed with "Cannot run range queries on datatype: RunEndEncoded(...)". Add the symmetric RunEndEncoded case, mirroring the existing Dictionary handling.

REE columns are rejected at plan time.

…arithmetic

thorfour · 2026-09-02T12:50:20Z

@kumarUjjawal anything else you require for this PR?

kumarUjjawal

Thank you @thorfour and @Jefffrey for the review.

## Which issue does this PR close? - Closes apache#26247. ## Rationale for this change `MIN` and `MAX` fail with an internal error on `RunEndEncoded` columns, while they work on `Dictionary` columns. See apache#26247 for a reproducer. ## What changes are included in this PR? `MIN`/`MAX` use a user-defined signature, so `get_min_max_result_type` alone decides the argument type. It unwraps `Dictionary` to its value type but passes `RunEndEncoded` through unchanged, so no cast is planned and the accumulator receives a type it has no implementation for. apache#24565 added `RunEndEncoded` handling to `coerced_from`, but user-defined signatures don't go through it. This PR adds a `RunEndEncoded` arm that recurses into the value type. With the recursion, `RunEndEncoded<_, Dictionary<K, V>>` coerces to `V`, the same result type `MAX` gives a plain `Dictionary<K, V>` column. Unwrapping one level would return the correct value, but type it as `Dictionary<K, V>`. The existing `Dictionary` arm unwraps only one level. I left it unchanged, because a dictionary whose values are themselves encoded seems rare. I'm happy to make it recursive in this PR if you'd prefer the two arms to match. ## What is the testing strategy for this PR? - `test_get_min_max_return_type_coerce_run_end_encoded` in `min_max.rs` covers `RunEndEncoded<Int32, UInt32>` → `UInt32` and `RunEndEncoded<Int32, Dictionary<UInt32, Utf8>>` → `Utf8`. - New cases in `run_end_encoded.slt` go through SQL planning, so they exercise the UDAF's coercion path rather than calling the accumulator directly: - ungrouped `MIN`/`MAX`; - their result types, via `arrow_typeof`; - grouped `MIN`/`MAX`; - `MAX` as a window function; - `MAX` over `RunEndEncoded<Int32, Dictionary<UInt32, Utf8>>`. Without the fix, the four `RunEndEncoded<Int32, UInt32>` cases fail with the internal error from apache#26247. ## Are there any user-facing changes? `MIN` and `MAX` now accept `RunEndEncoded` arguments and return the value type. There are no API changes. Generated-by: Claude Code (Claude Opus 5.5). I reviewed the change and can answer for it.

github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Aug 21, 2026

kumarUjjawal reviewed Aug 22, 2026

View reviewed changes

kumarUjjawal changed the title ~~Fix/date bin ree coercion~~ fix: support RunEndEncoded arrays in function and window frame type coercion Aug 22, 2026

Jefffrey reviewed Aug 22, 2026

View reviewed changes

Comment thread datafusion/expr/src/type_coercion/functions.rs Outdated

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Aug 24, 2026

thorfour mentioned this pull request Aug 27, 2026

Support RunEndEncoded arrays in arrow-arith numeric kernels (add/sub/mul/div) apache/arrow-rs#10891

Open

github-actions Bot removed the optimizer Optimizer rules label Aug 27, 2026

thorfour requested review from Jefffrey and kumarUjjawal August 27, 2026 16:27

kumarUjjawal reviewed Aug 27, 2026

View reviewed changes

Comment thread datafusion/sqllogictest/test_files/run_end_encoded.slt Outdated

thorfour force-pushed the fix/date-bin-ree-coercion branch from 2fd93a2 to 6663b37 Compare August 27, 2026 18:21

github-actions Bot added the optimizer Optimizer rules label Aug 27, 2026

thorfour requested a review from kumarUjjawal August 27, 2026 18:22

thorfour force-pushed the fix/date-bin-ree-coercion branch from 6663b37 to 223d6ca Compare August 27, 2026 18:28

thorfour added 5 commits August 27, 2026 13:30

cargo fmt

d7ca9e1

Added additional unit tests

da50b97

Removed window handling for REE and added unit test to ensure RANGE over

bf04547

REE columns are rejected at plan time.

github-actions Bot added physical-plan Changes to the physical-plan crate spark labels Aug 27, 2026

Re-add REE unwrapping in type coercion but reject in supports offset …

0d21a8d

…arithmetic

thorfour force-pushed the fix/date-bin-ree-coercion branch from 223d6ca to 0d21a8d Compare August 27, 2026 18:30

Jefffrey approved these changes Aug 29, 2026

View reviewed changes

kumarUjjawal approved these changes Sep 2, 2026

View reviewed changes

kumarUjjawal added this pull request to the merge queue Sep 2, 2026

Merged via the queue into apache:main with commit 3b63300 Sep 2, 2026
41 checks passed

yinli-systems mentioned this pull request Sep 17, 2026

Support RunEndEncoded arrays in numeric kernels apache/arrow-rs#11062

Open

thorfour deleted the fix/date-bin-ree-coercion branch September 17, 2026 17:20

nuno-faria added the v56.0.0 label Oct 3, 2026

This was referenced Oct 12, 2026

MIN/MAX fail with an internal error on RunEndEncoded columns #26247

Closed

fix: support RunEndEncoded arguments in MIN/MAX #26248

Merged

Conversation

thorfour commented Aug 21, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

thorfour commented Sep 2, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Aug 25, 2026 •

edited

Loading