You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
greatest/least compare struct arguments by field name instead of position when names differ only in case #6758
With spark.sql.caseSensitive=false, greatest and least return a different struct from Spark when the arguments' struct field names differ only in case.
Spark accepts such arguments because TypeCoercion.haveSameType compares the types ignoring case and nullability. It then compares the structs field by field by position, and the result keeps the first argument's field names. Comet passes the arguments through DataFusion's greatest/least coercion. When both structs hold the same set of field names (here x and X, in different orders), DataFusion's struct coercion matches the fields by name, and Arrow's struct cast reorders the second argument's values by name. So Comet compares different pairs of fields than Spark does.
This affects every supported Spark version (3.4 to 4.1), for float and non-float fields, for structs nested in arrays, and for three or more arguments. With three arguments that also differ in field nullability, the native cast fails instead: Cannot cast nullable struct field 'X' to non-nullable field.
Steps to reproduce
SETspark.sql.caseSensitive=false;
SELECT id, greatest(named_struct('x', CAST(id AS DOUBLE), 'X', 1D),
named_struct('X', 1D, 'x', CAST(id AS DOUBLE)))
FROM range(4);
With spark.comet.exec.range.enabled=true, spark.comet.sparkToColumnar.enabled=true, and spark.comet.sparkToColumnar.supportedOperatorList=Range, Comet runs this natively. For id = 0 Spark returns [1.0, 0.0] and Comet returns [0.0, 1.0]. least has the same problem: for id = 2 it returns [2.0, 1.0] where Spark returns [1.0, 2.0].
Expected behavior
Comet compares struct arguments by position, as Spark does, and returns [1.0, 0.0] for id = 0.
Additional context
coalesce, IF, CASE WHEN, array, map, concat, map_concat, array_append, array_insert, array_contains, array_intersect, array_except, arrays_overlap, =, <, IN and nullif match Spark for the same arguments. array_union with integer struct fields fails natively with Union is not implemented for 'ListArray' rather than returning a wrong answer.
Activity