Part of the Spark 4.2 unit-test enablement (GLUTEN-12569, PR #13163). This issue tracks re-enabling one root-cause group of UTs that are currently disabled to keep the gluten-ut/spark42 CI green.
Root cause
Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS
How it's disabled
.exclude(...) in gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala (GlutenDataFrameFunctionsSuite, GlutenDataFrameAggregateSuite).
Standard .exclude("<test>") entries.
Files updated
gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
Disabled tests (12)
GlutenDataFrameFunctionsSuite :: SPARK-54918: array set ops normalize -0.0 and NaN via VALUES inline tableGlutenDataFrameFunctionsSuite :: SPARK-54918: array_distinct normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite :: SPARK-54918: array_distinct normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite :: SPARK-54918: array_intersect normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite :: SPARK-54918: array_intersect normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite :: SPARK-54918: array_union normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite :: SPARK-54918: array_union normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite :: array_join with nullable nullReplacement under whole-stage codegenGlutenDataFrameAggregateSuite :: SPARK-55256: array_agg with RESPECT NULLS preserves nullsGlutenDataFrameAggregateSuite :: SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point typesGlutenDataFrameAggregateSuite :: SPARK-57298: collect_set normalizes NaN and -0.0 nested in complex typesGlutenDataFrameAggregateSuite :: collect_set with RESPECT NULLS preserves null in set
How to re-enable
Implement the underlying Gluten/Velox support, then remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and confirm the tests pass in the spark42 UT CI.
Related: PR #13163, GLUTEN-12569.
Umbrella: #13181
Part of the Spark 4.2 unit-test enablement (GLUTEN-12569, PR #13163). This issue tracks re-enabling one root-cause group of UTs that are currently disabled to keep the
gluten-ut/spark42CI green.Root cause
Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS
How it's disabled
.exclude(...)ingluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala(GlutenDataFrameFunctionsSuite, GlutenDataFrameAggregateSuite).Standard
.exclude("<test>")entries.Files updated
gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scalaDisabled tests (12)
GlutenDataFrameFunctionsSuite:: SPARK-54918: array set ops normalize -0.0 and NaN via VALUES inline tableGlutenDataFrameFunctionsSuite:: SPARK-54918: array_distinct normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite:: SPARK-54918: array_distinct normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite:: SPARK-54918: array_intersect normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite:: SPARK-54918: array_intersect normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite:: SPARK-54918: array_union normalizes -0.0 to +0.0GlutenDataFrameFunctionsSuite:: SPARK-54918: array_union normalizes -0.0 to +0.0 - literalsGlutenDataFrameFunctionsSuite:: array_join with nullable nullReplacement under whole-stage codegenGlutenDataFrameAggregateSuite:: SPARK-55256: array_agg with RESPECT NULLS preserves nullsGlutenDataFrameAggregateSuite:: SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point typesGlutenDataFrameAggregateSuite:: SPARK-57298: collect_set normalizes NaN and -0.0 nested in complex typesGlutenDataFrameAggregateSuite:: collect_set with RESPECT NULLS preserves null in setHow to re-enable
Implement the underlying Gluten/Velox support, then remove the corresponding
.exclude(...)/ allowlist /@Ignore/ignore(...)entries and confirm the tests pass in the spark42 UT CI.Related: PR #13163, GLUTEN-12569.
Umbrella: #13181