You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Three exclusions in dev/diffs/*.diff cite Comet issues that have since been fixed, so they now suppress Spark SQL test coverage for no reason. Each is present in more than one version diff, so they should be removed across all affected diffs in one pass rather than in whichever version PR happens to touch them next: new version diffs are seeded from the previous version's, so a partial fix diverges the diffs and the next reseed reintroduces the stale entry.
fix: fall back for shredded Variant scans on Spark 4.0 #4084 added the shredded-Variant scan fallback and already removed these same exclusions from dev/diffs/4.0.4.diff (it has zero occurrences of VariantShreddingSuite); 4.1.3.diff was never cleaned up.
The related SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded strings test in ParquetIOSuite is still tagged IgnoreComet("Comet doesn't support DELTA encoding yet") and should be re-enabled with it.
Affected: dev/diffs/4.0.4.diff, dev/diffs/4.1.3.diff, dev/diffs/4.2.0.diff (byte-identical in all three).
If the legacy native_comet scan still cannot read these encodings, scope the exclusion to that scan implementation rather than removing the whole matrix.
Suggested approach
One PR per item, each removing the exclusion from every diff that carries it, verified by the corresponding Spark SQL Tests CI job for each affected version. Where a specific case still fails, replace the blanket exclusion with a case-scoped IgnoreComet linked to a current issue, per the rule in docs/source/contributor-guide/adding_a_new_spark_version.md ("do not disable whole suites").
Three exclusions in
dev/diffs/*.diffcite Comet issues that have since been fixed, so they now suppress Spark SQL test coverage for no reason. Each is present in more than one version diff, so they should be removed across all affected diffs in one pass rather than in whichever version PR happens to touch them next: new version diffs are seeded from the previous version's, so a partial fix diverges the diffs and the next reseed reintroduces the stale entry.Surfaced in review of #4950 by @sunchao.
1. Variant shredding suites
VariantShreddingSuiteandParquetVariantShreddingSuiteare given anIgnoreCometSuitemixin, which skips all 15 tests whenENABLE_COMET=true.VariantShreddingSuitetest failures with Spark 4.0.0 #2209, closed by merged PR fix: fall back for shredded Variant scans on Spark 4.0 #4084.dev/diffs/4.0.4.diff(it has zero occurrences ofVariantShreddingSuite);4.1.3.diffwas never cleaned up.dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff.IgnoreCometSuiteimport in each file.This is the highest-value one: it drops regression coverage for the silent-null corruption that #4084 fixed.
2.
having-and-order-by-recursive-type-name-resolution.sqlThe input file carries a
--SET spark.comet.enabled = falseheader, which runs all 45 queries without Comet.Struct(Map(...))HAVING / ORDER BY shape.dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff.3. Parquet V2 writer matrix in
ParquetTypeWideningSuitewithAllParquetWriters { ... }is commented out, leaving all 82 generated widening cases on Parquet's default V1 writer.native_datafusionandnative_iceberg_compatscans supportDELTA_BINARY_PACKED,DELTA_BYTE_ARRAYandDELTA_LENGTH_BYTE_ARRAY.SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded stringstest inParquetIOSuiteis still taggedIgnoreComet("Comet doesn't support DELTA encoding yet")and should be re-enabled with it.dev/diffs/4.0.4.diff,dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff(byte-identical in all three).native_cometscan still cannot read these encodings, scope the exclusion to that scan implementation rather than removing the whole matrix.Suggested approach
One PR per item, each removing the exclusion from every diff that carries it, verified by the corresponding
Spark SQL TestsCI job for each affected version. Where a specific case still fails, replace the blanket exclusion with a case-scopedIgnoreCometlinked to a current issue, per the rule indocs/source/contributor-guide/adding_a_new_spark_version.md("do not disable whole suites").