You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) #6768
From Spark 4.0.5, 4.1.4 and 4.2.1, arrays_overlap treats -0.0 and 0.0 as one value in flat
float and double arrays. SPARK-59602 handles "primitive floating-point values locally in arrays_overlap". Comet's native spark_arrays_overlap keeps them apart in its flat path, as
earlier Spark releases do (see the note on arrays_overlap_list_generic in arrays_overlap.rs),
so on those releases it returns false where Spark returns true.
Spark 4.2.0 normalized these arguments in the plan (SPARK-54918), so Comet received normalized
input there. SPARK-59602 replaced that rewrite in 4.2.1.
Steps to reproduce
Build Comet against the staged v4.1.4-rc2 (orgapachespark-1533), v4.0.5-rc1
(orgapachespark-1534) or v4.2.1-rc1 (orgapachespark-1531) artifacts and run expressions/array/arrays_overlap.sql:
SELECT a, b, arrays_overlap(a, b) FROM test_overlap_dbl
Spark returns [List(0.0),List(-0.0),true] and Comet [List(0.0),List(-0.0),false]. The arrays_overlap [double] and [float] cases of CometFloatSemanticsSuite fail on the same
zero pairs. main at f80042f fails the same way.
Expected behavior
Comet matches each version: flat signed zeros stay apart before 4.0.5, 4.1.4 and 4.2, and merge
from 4.0.5, 4.1.4 and 4.2.1 (4.2.0 receives them already normalized). #6563 adds the same version
check for array_distinct and array_union (ArraySetSupport.normalizesFloats), and the kernel
can take the result as a flag.
Additional context
Found by running the array suites against the Spark release candidates for #6563. Part of #6385.
Activity
kazuyukitanimura commentedon Oct 12, 2026
This can be the bug we fixed in Spark