Skip to content

arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) #6768

Open

Description

@andygrove

Describe the bug

From Spark 4.0.5, 4.1.4 and 4.2.1, arrays_overlap treats -0.0 and 0.0 as one value in flat
float and double arrays. SPARK-59602 handles "primitive floating-point values locally in
arrays_overlap". Comet's native spark_arrays_overlap keeps them apart in its flat path, as
earlier Spark releases do (see the note on arrays_overlap_list_generic in arrays_overlap.rs),
so on those releases it returns false where Spark returns true.

Spark 4.2.0 normalized these arguments in the plan (SPARK-54918), so Comet received normalized
input there. SPARK-59602 replaced that rewrite in 4.2.1.

Steps to reproduce

Build Comet against the staged v4.1.4-rc2 (orgapachespark-1533), v4.0.5-rc1
(orgapachespark-1534) or v4.2.1-rc1 (orgapachespark-1531) artifacts and run
expressions/array/arrays_overlap.sql:

SELECT a, b, arrays_overlap(a, b) FROM test_overlap_dbl

Spark returns [List(0.0),List(-0.0),true] and Comet [List(0.0),List(-0.0),false]. The
arrays_overlap [double] and [float] cases of CometFloatSemanticsSuite fail on the same
zero pairs. main at f80042f fails the same way.

Expected behavior

Comet matches each version: flat signed zeros stay apart before 4.0.5, 4.1.4 and 4.2, and merge
from 4.0.5, 4.1.4 and 4.2.1 (4.2.0 receives them already normalized). #6563 adds the same version
check for array_distinct and array_union (ArraySetSupport.normalizesFloats), and the kernel
can take the result as a flag.

Additional context

Found by running the array suites against the Spark release candidates for #6563. Part of #6385.

Activity

  1. added
    priority:criticalData corruption, silent wrong results, security issues
    and removed on Oct 12, 2026
  2. kazuyukitanimura commented on Oct 12, 2026

    @kazuyukitanimura
    Contributor

    This can be the bug we fixed in Spark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingpriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions