Skip to content

Spark 4.x: Tracking disabled test suites #11550

Open
4 of 9 issues completed
Open
Spark 4.x: Tracking disabled test suites#11550
4 of 9 issues completed
#11725 (+2)
Labels
bugSomething isn't workingtrackerTracker of issues in the same categorytriage

Description

@baibaichen
baibaichen
opened on Feb 3, 2026

Please leave a comment and mark on the TODO tests in this issue if you want to contribute

Backend

VL(Velox)

Bug Description

These suites were commented out in #11512 with TODO markers(and were updated based on #11800) indicating they need to be fixed and re-enabled.

Context

The test suites below have been disabled due to various failures. Each table shows the status for both Spark 4.0 and 4.1:

  • 🔴 = Suite is commented out (disabled)
  • 🟢 = Suite is enabled

The failure count (when available) is shown in the status column.

org.apache.spark.sql

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenDataFrameSubquerySuite 🟢 🟢 @baibaichen Fixed in #11727
GlutenExplainSuite 🟢 🟢 Disabled in #12546
GlutenJoinHintSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenLogQuerySuite 🟢 🟢 @kapilks Fixed
GlutenRandomDataGeneratorSuite 🟢 🟢 #11919 ixed in #12533
GlutenSetCommandSuite 🟢 🟢 Fixed in #12533
GlutenSingleLevelAggregateHashMapSuite 🟢 🟢 @kapilks Fixed
GlutenSparkSessionJobTaggingAndCancellationSuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenTwoLevelAggregateHashMapSuite 🟢 🟢 @kapilks Fixed
GlutenTwoLevelAggregateHashMapWithVectorizedMapSuite 🟢 🟢 @kapilks Fixed
GlutenVariantEndToEndSuite 🟢 🟢 @baibaichen Fixed in #11726
GlutenVariantShreddingSuite 🟢 🟢 @baibaichen Fixed in #11726
GlutenXmlFunctionsSuite 🟢 🟢 @baibaichen Fixed in #11725
GlutenSparkSessionExtensionSuite 🟢 🟢 Fixed in #12475
GlutenTPCDSV1_4_PlanStabilitySuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCDSV1_4_PlanStabilityWithStatsSuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCDSV2_7_PlanStabilitySuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCDSV2_7_PlanStabilityWithStatsSuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCDSModifiedPlanStabilitySuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCDSModifiedPlanStabilityWithStatsSuite 🟢 🟢 @baibaichen Fixed in #11799
GlutenTPCHPlanStabilitySuite 🟢 🟢 @baibaichen Fixed in #11799

catalyst.expressions

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenCastWithAnsiOnSuite 🔴 (4 failures) 🔴 (10 failures) #10134 ANSI + TimeType
GlutenCollationRegexpExpressionsSuite 🟢 🔴 (1 failure) #11913 Velox split limit unsupported
GlutenCsvExpressionsSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenExpressionEvalHelperSuite 🟢 🟢 Disabled in #12484
GlutenObjectExpressionsSuite 🟢 🟢 Disabled in #12484
GlutenOrderingSuite 🟢 🟢 #11919 Fixed in #12484
GlutenScalaUDFSuite 🟢 🟢 Disabled in #12484
GlutenToPrettyStringSuite 🟢 🟢 #11918 Fixed in #12048
GlutenXmlExpressionsSuite 🟢 🟢 @baibaichen Fixed in #11580

connector

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenGroupBasedUpdateTableSuite 🟢 🟢 #11912 Fixed in #12493
GlutenMergeIntoDataFrameSuite 🟢 🟢 @baibaichen Fixed in #11812

execution

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenColumnarRulesSuite 🟢 🟢 #11920 Fixed in #12004
GlutenDataSourceScanExecRedactionSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenDataSourceV2ScanExecRedactionSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenExternalAppendOnlyUnsafeRowArraySuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenHiveResultSuite 🟢 🟢 #11919 Fixed in #12484
GlutenInsertSortForLimitAndOffsetSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenLogicalPlanTagInSparkPlanSuite 🟢 🟢 @baibaichen Fixed in #11833. Core fix: propagate LOGICAL_PLAN_TAG during offload
GlutenMultiStatefulOperatorsSuite 🟢 🟢 #11911 Fixed in #11957
GlutenPlannerSuite 🟢 🟢 Disabled in #12512
GlutenProjectedOrderingAndPartitioningSuite 🟢 🟢 Disabled in #12512
GlutenRemoveRedundantProjectsSuite 🟢 🟢 Fixed in #12506
GlutenRemoveRedundantSortsSuite 🟢 🟢 Fixed in #12497
GlutenSQLExecutionSuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenSQLJsonProtocolSuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenShufflePartitionsUtilSuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenSimpleSQLViewSuite 🟢 🟢 #11912 Fixed in #12502
GlutenSparkPlanSuite 🟢 🟢 @baibaichen Fixed in #11816
GlutenUnsafeRowSerializerSuite 🟢 🟢 @baibaichen Fixed in #11847
GlutenWholeStageCodegenSparkSubmitSuite 🟢 🟢 Disabled in #12512
GlutenWholeStageCodegenSuite 🟢 🟢 Fixed in #12507

execution.joins

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenSingleJoinSuite 🟢 🟢 @baibaichen Fixed in #11577

execution.datasources.parquet

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenParquetTypeWideningSuite 🟢 🟢 @baibaichen Fixed in #11683
GlutenParquetVariantShreddingSuite 🟢 🟢 @baibaichen Fixed in #11726

execution.datasources.text

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenWholeTextFileV1Suite 🟢 🟢 @baibaichen Fixed in #11816
GlutenWholeTextFileV2Suite 🟢 🟢 @baibaichen Fixed in #11816

execution.python

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenPythonDataSourceSuite 🔴 (1 failures) 🔴 @baibaichen #11908
GlutenPythonUDFSuite 🔴 (1 failures) 🔴 @baibaichen #11908
GlutenPythonUDTFSuite 🔴 🔴 @baibaichen #11908
GlutenRowQueueSuite 🟢 🔴 @baibaichen #11908

sources

Suite Spark 4.0 Spark 4.1 Owner Comments
GlutenBucketedReadWithHiveSupportSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenBucketedWriteWithHiveSupportSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenCommitFailureTestRelationSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenJsonHadoopFsRelationSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenParquetHadoopFsRelationSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)
GlutenSimpleTextHadoopFsRelationSuite 🟢 🟢 @baibaichen Deleted in #11847 (TestHiveSingleton conflict)

streaming

Structured Streaming suites (20 disabled) are tracked in #11911.

Summary Statistics

  • Total suites disabled in Spark 4.0: 69
  • Total suites disabled in Spark 4.1: 79
  • Unique suites across both versions: 79

PRs

PR Status Suites fixed
#11577 ✅ Merged 1: SingleJoinSuite
#11580 ✅ Merged 1: XmlExpressionsSuite
#11683 ✅ Merged 1: ParquetTypeWideningSuite
#11725 ✅ Merged 1: XmlFunctionsSuite
#11726 ✅ Merged 2: VariantEndToEnd, VariantShreddingSuite
#11727 ✅ Merged 1: DataFrameSubquerySuite
#11799 ✅ Merged 7: TPCDS PlanStability suites
#11812 ✅ Merged 1: MergeIntoDataFrame (JobTagging reverted in #11908)
#11833 ✅ Merged 1: LogicalPlanTagInSparkPlan (150 tests)
#11847 ✅ Merged 6 enabled + 7 deleted: SystemPropertyTrait suites
#11816 ✅ Merged 8 suites + 2 trait fix + 1 disable
#11908 🔵 Draft 4: PythonDataSource, PythonUDF, PythonUDTF, RowQueue

Activity

  1. baibaichen
    added
    bugSomething isn't working
    triage
    on Feb 3, 2026
  2. baibaichen
    added a parent issue on Feb 3, 2026
  3. github-project-automation
    added this to Gluten Velox Devon Feb 3, 2026
  4. baibaichen
    changed the title [-]Spark 4.0/4.1: Tracking disabled test suites (commit 4e3df671)[/-] [+]Spark 4.x: Tracking disabled test suites[/+] on Feb 3, 2026
  5. FelixYBW
    added
    trackerTracker of issues in the same category
    on Feb 4, 2026

  6. kapilks commented on Feb 4, 2026

    @kapilks
    kapilks
    on Feb 4, 2026
    Contributor

    Can work on GlutenLogQuerySuite

  7. baibaichen commented on Feb 5, 2026

    @baibaichen
    baibaichen
    on Feb 5, 2026
    ContributorAuthor

    Can work on GlutenLogQuerySuite
    Thanks @kapilks

  8. Surbhi-Vijay commented on Feb 5, 2026

    @Surbhi-Vijay
    Surbhi-Vijay
    on Feb 5, 2026
    Contributor

    I'll take:

    GlutenExplainSuite
    GlutenSetCommandSuite
    GlutenGroupBasedUpdateTableSuite
    GlutenSparkPlanSuite
    GlutenWholeStageCodegenSuite
    GlutenBucketedReadWithHiveSupportSuite
    GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite

  9. kapilks commented on Feb 5, 2026

    @kapilks
    kapilks
    on Feb 5, 2026
    Contributor

    I can take these:
    GlutenSingleLevelAggregateHashMapSuite
    GlutenTwoLevelAggregateHashMapSuite
    GlutenTwoLevelAggregateHashMapWithVectorizedMapSuite

  10. baibaichen commented on Feb 6, 2026

    @baibaichen
    baibaichen
    on Feb 6, 2026
    ContributorAuthor

    Claiming GlutenSingleJoinSuite - fixed in PR #11577

  11. baibaichen
    closed this as completedin #11577on Feb 6, 2026
  12. github-project-automation
    moved this to Done in Gluten Velox Devon Feb 6, 2026
  13. baibaichen
    reopened this on Feb 10, 2026

  14. kapilks commented on Feb 11, 2026

    @kapilks
    kapilks
    on Feb 11, 2026
    Contributor

    Will take GlutenInsertSortForLimitAndOffsetSuite

  15. baibaichen
    closed this as completedin #11580on Feb 28, 2026
  16. baibaichen
    reopened this on Mar 2, 2026
  17. baibaichen
    added a sub-issue on Mar 3, 2026
  18. baibaichen
    closed this as completedin #11725on Mar 12, 2026
  19. baibaichen
    reopened this on Mar 13, 2026

  20. baibaichen commented on Mar 30, 2026

    @baibaichen
    baibaichen
    on Mar 30, 2026
    ContributorAuthor

    Update: PR #11847 (draft) adds GlutenTestSetWithSystemPropertyTrait and enables 6 suites (GlutenSQLExecutionSuite, GlutenSQLJsonProtocolSuite, GlutenShufflePartitionsUtilSuite, GlutenExternalAppendOnlyUnsafeRowArraySuite, GlutenUnsafeRowSerializerSuite, GlutenSparkSessionJobTaggingAndCancellationSuite). Also removes 7 TestHiveSingleton suites that cannot load GlutenPlugin.

  21. baibaichen
    added sub-issues on Apr 11, 2026

  22. baibaichen commented on Apr 11, 2026

    @baibaichen
    baibaichen
    on Apr 11, 2026
    ContributorAuthor

    Triage update (2026-04-11)

    Completed triage of all remaining TODO suites. Changes:

    Failure count corrections:

    • GlutenExplainSuite: 1 → 4 failures
    • GlutenPlannerSuite: 1 → 15 failures
    • GlutenRemoveRedundantSortsSuite: 1 → 5 failures

    Owner assignments (sub-issues created):

    Released for community (RC):

    • GlutenExplainSuite, GlutenPlannerSuite, GlutenProjectedOrderingAndPartitioningSuite, GlutenRemoveRedundantProjectsSuite, GlutenRemoveRedundantSortsSuite — owner cleared, available for contribution

    Streaming table moved to #11911 — 20 disabled suites now tracked there.

    New feature suites tracked under #11910 (Spark 4.x feature tracking):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtrackerTracker of issues in the same categorytriage

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions