Skip to content

[Spark 4.2 UT] Umbrella: re-enable disabled Spark 4.2 unit tests (GLUTEN-12569) #13181

Description

@akshaytayal

Umbrella tracking issue for re-enabling the Spark 4.2 unit tests that are currently disabled to keep the gluten-ut/spark42 CI green.

Added in PR #13163 (GLUTEN-12569). 150 UTs are disabled across 7 root-cause themes. Each theme has its own sub-issue with the exact disabled tests, the disable mechanism, and the files updated. As the underlying Gluten/Velox support lands, remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and close the sub-issue.

Sub-issues (7)

Summary

# Theme Items Sub-issue
1 Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten 29 #13174
2 Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path 39 #13175
3 Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS 12 #13176
4 Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) 46 #13177
5 Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) 6 #13178
6 Spark 4.2 framework/API changes breaking Gluten tests (non-velox) 7 #13179
7 File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) 11 #13180
Total 150

How these were disabled

  • .exclude(...) in gluten-ut/spark42/.../velox/VeloxTestSettings.scala (most suites, honoring BackendTestSettings.shouldRun)
  • allowlist removal in gluten-ut/spark42/.../velox/VeloxSQLQueryTestSettings.scala (GlutenSQLQueryTestSuite .sql golden tests — that suite ignores the exclude registry)
  • class-level @org.scalatest.Ignore for whole-suite aborts (GlutenBloomFilterFallbackSuite, MiscOperatorSuite, GlutenWholeStageCodegenSuite)
  • test(...)->ignore(...) for one standalone test (GlutenAutoAdjustStageResourceProfileSuite)

All disables are CI-confirmed (local-only false-positives were removed via a validation run with excludes reverted).

Related: PR #13163, GLUTEN-12569.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions