Background
PR #5166 gates collect_set input normalization on Spark 4.2+, where Spark introduced the normalization contract.
On Spark 3.4 through 4.1, normalizing the input would make Comet collapse -0.0 and 0.0 into one entry, matching Spark. The existing NaN incompatibility would remain because pre-4.2 Spark keeps distinct NaN entries while Comet deduplicates them.
Trade-off
For array-typed children, Spark's recursive normalizer produces ArrayTransform, which Comet executes through the JVM codegen dispatcher. Removing the version gate would add that cost on older Spark versions too. If spark.comet.exec.scalaUDF.codegen.enabled=false, the aggregate safely falls back to Spark.
Task
Evaluate removing or narrowing the isSpark42Plus gate so signed-zero behavior matches Spark 3.4 through 4.1 without unnecessarily routing nested arrays through JVM codegen. Add regression coverage and update the compatibility wording if the signed-zero divergence is removed.
Related: #5166 and review discussion.
Background
PR #5166 gates
collect_setinput normalization on Spark 4.2+, where Spark introduced the normalization contract.On Spark 3.4 through 4.1, normalizing the input would make Comet collapse
-0.0and0.0into one entry, matching Spark. The existing NaN incompatibility would remain because pre-4.2 Spark keeps distinct NaN entries while Comet deduplicates them.Trade-off
For array-typed children, Spark's recursive normalizer produces
ArrayTransform, which Comet executes through the JVM codegen dispatcher. Removing the version gate would add that cost on older Spark versions too. Ifspark.comet.exec.scalaUDF.codegen.enabled=false, the aggregate safely falls back to Spark.Task
Evaluate removing or narrowing the
isSpark42Plusgate so signed-zero behavior matches Spark 3.4 through 4.1 without unnecessarily routing nested arrays through JVM codegen. Add regression coverage and update the compatibility wording if the signed-zero divergence is removed.Related: #5166 and review discussion.