Skip to content

greatest/least compare struct arguments by field name instead of position when names differ only in case #6758

Closed

Description

@viirya

Describe the bug

With spark.sql.caseSensitive=false, greatest and least return a different struct from Spark when the arguments' struct field names differ only in case.

Spark accepts such arguments because TypeCoercion.haveSameType compares the types ignoring case and nullability. It then compares the structs field by field by position, and the result keeps the first argument's field names. Comet passes the arguments through DataFusion's greatest/least coercion. When both structs hold the same set of field names (here x and X, in different orders), DataFusion's struct coercion matches the fields by name, and Arrow's struct cast reorders the second argument's values by name. So Comet compares different pairs of fields than Spark does.

This affects every supported Spark version (3.4 to 4.1), for float and non-float fields, for structs nested in arrays, and for three or more arguments. With three arguments that also differ in field nullability, the native cast fails instead: Cannot cast nullable struct field 'X' to non-nullable field.

Steps to reproduce

SET spark.sql.caseSensitive=false;
SELECT id, greatest(named_struct('x', CAST(id AS DOUBLE), 'X', 1D),
                    named_struct('X', 1D, 'x', CAST(id AS DOUBLE)))
FROM range(4);

With spark.comet.exec.range.enabled=true, spark.comet.sparkToColumnar.enabled=true, and spark.comet.sparkToColumnar.supportedOperatorList=Range, Comet runs this natively. For id = 0 Spark returns [1.0, 0.0] and Comet returns [0.0, 1.0]. least has the same problem: for id = 2 it returns [2.0, 1.0] where Spark returns [1.0, 2.0].

Expected behavior

Comet compares struct arguments by position, as Spark does, and returns [1.0, 0.0] for id = 0.

Additional context

coalesce, IF, CASE WHEN, array, map, concat, map_concat, array_append, array_insert, array_contains, array_intersect, array_except, arrays_overlap, =, <, IN and nullif match Spark for the same arguments. array_union with integer struct fields fails natively with Union is not implemented for 'ListArray' rather than returning a wrong answer.

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions