Skip to content

[Bug] array_min and array_max differ from Spark on signed-zero ties #5401

Description

@sunchao

Describe the bug

array_min and array_max do not preserve Spark's choice of the first element when +0.0 and -0.0 compare equal. The corrected fixtures in #5393 expose this for array_min. The reverse zero order also exposes it for array_max.

At upstream commit 91f9fbed24ee28f99b23ed559e80e818c9696044, both expressions are classified as compatible without checking spark.comet.exec.strictFloatingPoint. They therefore use the differing native implementation even when strict floating-point behavior is requested.

Reproduction

Use a Parquet column so the extrema are evaluated at runtime:

SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SET spark.comet.exec.strictFloatingPoint=true;

CREATE TABLE comet_extrema_zero_repro (id INT, arr ARRAY<DOUBLE>) USING parquet;
INSERT INTO comet_extrema_zero_repro VALUES
  (1, array(double('0.0'), double('-0.0'))),
  (2, array(double('-0.0'), double('0.0')));

SELECT id, array_min(arr), array_max(arr)
FROM comet_extrema_zero_repro ORDER BY id;

The same issue occurs for ARRAY<FLOAT>.

Input Spark min Native min Spark max Native max
[+0.0, -0.0] +0.0 -0.0 +0.0 +0.0
[-0.0, +0.0] -0.0 -0.0 -0.0 +0.0

Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both floating-point widths and on both sides of the array-length-32 kernel boundary.

Expected behavior

Strict floating-point mode should use Spark-compatible evaluation. The native implementation should eventually preserve Spark's first-equal-element behavior as well. A narrow first fix can route strict-mode extrema through the existing Spark-codegen fallback, while keeping this issue open for native parity.

Relevant code: CometArrayMin and CometArrayMax in spark/src/main/scala/org/apache/comet/serde/arrays.scala. This is a follow-up to #5271 and #5393, not a regression introduced by those fixture changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions