Describe the bug
array_min and array_max do not preserve Spark's choice of the first element when +0.0 and -0.0 compare equal. The corrected fixtures in #5393 expose this for array_min. The reverse zero order also exposes it for array_max.
At upstream commit 91f9fbed24ee28f99b23ed559e80e818c9696044, both expressions are classified as compatible without checking spark.comet.exec.strictFloatingPoint. They therefore use the differing native implementation even when strict floating-point behavior is requested.
Reproduction
Use a Parquet column so the extrema are evaluated at runtime:
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SET spark.comet.exec.strictFloatingPoint=true;
CREATE TABLE comet_extrema_zero_repro (id INT, arr ARRAY<DOUBLE>) USING parquet;
INSERT INTO comet_extrema_zero_repro VALUES
(1, array(double('0.0'), double('-0.0'))),
(2, array(double('-0.0'), double('0.0')));
SELECT id, array_min(arr), array_max(arr)
FROM comet_extrema_zero_repro ORDER BY id;
The same issue occurs for ARRAY<FLOAT>.
| Input |
Spark min |
Native min |
Spark max |
Native max |
[+0.0, -0.0] |
+0.0 |
-0.0 |
+0.0 |
+0.0 |
[-0.0, +0.0] |
-0.0 |
-0.0 |
-0.0 |
+0.0 |
Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both floating-point widths and on both sides of the array-length-32 kernel boundary.
Expected behavior
Strict floating-point mode should use Spark-compatible evaluation. The native implementation should eventually preserve Spark's first-equal-element behavior as well. A narrow first fix can route strict-mode extrema through the existing Spark-codegen fallback, while keeping this issue open for native parity.
Relevant code: CometArrayMin and CometArrayMax in spark/src/main/scala/org/apache/comet/serde/arrays.scala. This is a follow-up to #5271 and #5393, not a regression introduced by those fixture changes.
Describe the bug
array_minandarray_maxdo not preserve Spark's choice of the first element when+0.0and-0.0compare equal. The corrected fixtures in #5393 expose this forarray_min. The reverse zero order also exposes it forarray_max.At upstream commit
91f9fbed24ee28f99b23ed559e80e818c9696044, both expressions are classified as compatible without checkingspark.comet.exec.strictFloatingPoint. They therefore use the differing native implementation even when strict floating-point behavior is requested.Reproduction
Use a Parquet column so the extrema are evaluated at runtime:
The same issue occurs for
ARRAY<FLOAT>.[+0.0, -0.0]+0.0-0.0+0.0+0.0[-0.0, +0.0]-0.0-0.0-0.0+0.0Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both floating-point widths and on both sides of the array-length-32 kernel boundary.
Expected behavior
Strict floating-point mode should use Spark-compatible evaluation. The native implementation should eventually preserve Spark's first-equal-element behavior as well. A narrow first fix can route strict-mode extrema through the existing Spark-codegen fallback, while keeping this issue open for native parity.
Relevant code:
CometArrayMinandCometArrayMaxinspark/src/main/scala/org/apache/comet/serde/arrays.scala. This is a follow-up to #5271 and #5393, not a regression introduced by those fixture changes.