[Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline #12743

The Delta Spark UT pipeline landed in #12388 (fixes #9296). It runs delta-io/delta's spark test suite against a Gluten Velox bundle and gates each run against a committed baseline of known failures.

The committed main baseline contains 546 known-failure entries as of 2026-10-15 (baseline at 020211a4b). The verified reductions since the 716-entry baseline are listed below. This issue tracks the work to bring that number down and to improve the pipeline itself. It is a tracking issue only — discussion belongs in the linked issues.

1. Correctness gaps behind the known-failure baseline

These are not a consequence of Gluten declining to offload: a plan Gluten cannot offload should fall back to vanilla Spark and the test should still pass. Most entries below are bugs -- fallback not happening, metrics differing from vanilla, or the native side failing. A separate group is expected rather than fixable here: tests that assert on the query plan see a different plan once Gluten offloads, which is by design. Those are worth splitting out during triage, since they should either be fixed upstream (as delta-io/delta#7104 and delta-io/delta#7105 were) or stay in the baseline permanently.

Merged baseline reductions since #13058

The file history accounts for the full 170-entry reduction (716 to 546). Counts below follow actual merge order on main, rather than pre-merge totals quoted in individual PR descriptions.

The extra data-skipping case removed by #13252 was observed passing alongside the 54 MERGE fixes; this does not establish the same root cause. #12967 fixes the NTZ statistics trigger, not the general unsupported-plan fallback requested in #12538. The remaining column-mapping footer-metadata failure exposed by #13134 is tracked in #13288.

Current tracked clusters

Recounted from the same 546-entry baseline at 020211a4b; this is not an exhaustive list of remaining failures:

Historical triage also identified overlapping cross-suite clusters: 225 baseline tests passed with IncrementMetric offload disabled (#9003; measured below, replacing the earlier ~226 estimate), and ~47 were attributed to the Delta stats tracker ClassCastException when the stats plan could not be offloaded (#12538; estimate from the triage in #12388). These are historical measurements, not additional current failure counts; do not add them to the table totals.

IncrementMetric offload experiment (2026-09-23)

Diagnostic draft #13112 (5e18f8d5b) disables Velox's IncrementMetric filter/project offload while retaining native Delta scans. The completed Delta Spark UT run tested Spark 4.1 / Delta v4.2.0 against the unchanged 716-entry baseline, with no quarantine or test-selection changes.

The 225 now-passing tests span 55 suites: 128 in the two DescribeDeltaHistory suites (64 each), 92 in generated MERGE suites, and 5 in Delta CDC streaming suites. Of the 130 history-suite baseline failures, only operation metrics - merge still fails in each suite (2 entries); those need further triage.

The aggregate job succeeded. The workflow is red because the gate deliberately fails on now-passing baseline tests, not because this experiment introduced new failures. This measures the impact of disabling offload; it is not a native IncrementMetric fix. This experiment alone does not justify closing #9003 or removing its baseline entries; do not replace the committed baseline with this diagnostic run's 491-entry artifact. The 716-entry starting point is historical, while the current baseline changes as independently validated fixes land.

Related type-support gap: TIMESTAMP_NTZ (#11622), which is one trigger of #12538.

• [VL] IncrementMetric doesn't work in some cases #9003 — IncrementMetric: 225 baseline tests now pass in diagnostic [GLUTEN-9003][VL] Disable IncrementMetric offload for Delta CI investigation #13112; a production fix is still needed.

• [VL][Delta] Gluten Delta stats tracker should fall back when the stats plan can't be offloaded (ClassCastException on TIMESTAMP_NTZ) #12538 — General fallback for unsupported Delta statistics plans remains open. [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics #12967 fixed the TIMESTAMP_NTZ min/max trigger and removed 42 baseline entries, but did not implement arbitrary stats-plan fallback.

• [VL] Support TIMESTAMP_NTZ Type #11622 — TIMESTAMP_NTZ support

• Hadoop file system options doesn't work #4963 — Hadoop file system options

• Triage the 2 DescribeDeltaHistory failures remaining in the offload-disabled experiment, plus the remaining ImplicitMergeCasting and DataSkipping clusters, and file one issue per root cause. The 42 previously listed RowTrackingBackfill cases were fixed by [GLUTEN-12743][VL] Preserve transaction observers in native Delta commands #13134.

• [GLUTEN-12743][CI] Make Delta DV row-group fixtures deterministic #13036 — Made the DeltaParquetFileFormatSuite row-group fixture deterministic for native writes; 9 verified passing tests were removed from the baseline (707 → 698).

• [GLUTEN-12743][CI] Stabilize Delta predicate-pushdown fixture #13058 — Made the DeletionVectorsWithPredicatePushdownSuite row-group fixture deterministic while retaining native writes. All 50 suite tests execute: 30 pass and 20 downstream failures remain. Replaced the two obsolete suite-abort entries with those 20 failures (698 → 716); this exposes previously blocked coverage rather than introducing 18 regressions.

• Triage the remaining DeletionVectorsWithPredicatePushdownSuite failures exposed by [GLUTEN-12743][CI] Stabilize Delta predicate-pushdown fixture #13058: 12 missing-pushed-filter assertions and 6 split-count mismatches. Keep these in the baseline until fixes are verified; the 2 inherited 2B-row OOM force-failures remain tracked under [VL] Native memory OOMs the test/executor JVM when reading large (billions-of-rows) Delta tables with deletion vectors #12387.

• [GLUTEN-12743][VL] Fall back Delta scans requiring generated DV metadata #13042 — Merged 2026-10-15: scoped JVM fallback fixes generated Delta row-index/null and deleted-row-flag failures when useMetadataRowIndex=false. Removed nine now-passing DeltaParquetFileFormatSuite baseline entries after Delta CI passed all eight shards. Ordinary scans and Spark's _metadata.row_index remain eligible for native offload.

• [VL][Delta] Missing Spark schema metadata in native Parquet footer #13288 — Investigate the remaining native Parquet footer schema-metadata failure exposed by [GLUTEN-12743][VL] Preserve transaction observers in native Delta commands #13134; its baseline entry remains listed.

2. Native bugs currently worked around in CI

Temporary workarounds should be removed together with the corresponding fixes. Force-failed tests remain visible in the known-failure baseline; they are not silently skipped.

• [VL] Native Delta DV bitmap aggregator aborts on a Long.MAX_VALUE sentinel row index during MERGE with deletion vectors (intermittent VeloxRuntimeError) #12377 — DV bitmap aggregator aborts on an invalid row index. Fixed by [GLUTEN-6887][VL] Daily Update Velox Version (2026_08_21) #12846; the two flaky-error-patterns.txt quarantine entries were removed in [GLUTEN-12377][CI] Stop quarantining the Delta DV row-index failures #12829.
• [VL] Native memory OOMs the test/executor JVM when reading large (billions-of-rows) Delta tables with deletion vectors #12387 — native memory OOM when reading billions-of-rows tables. Two 2B-row test bodies in DeletionVectorsSuite are force-failed in apply-delta-test-patches.sh (called by setup-delta.sh) to stop them OOM-killing the shard. These also execute in DeletionVectorsWithPredicatePushdownSuite now that [GLUTEN-12743][CI] Stabilize Delta predicate-pushdown fixture #13058 fixes its setup. Remove that patch when fixed. PR: [GLUTEN-12387][VL] Re-enable large deletion vector tests #12866
• Unsupported operations should induce a fallback for Delta plans rather than failing (raised by @zhztheplayer in [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline #12388; [VL][Delta] Gluten Delta stats tracker should fall back when the stats plan can't be offloaded (ClassCastException on TIMESTAMP_NTZ) #12538 is the concrete instance)

3. Pipeline improvements

• Automatically remove now-passing tests from the Delta baseline. [MINOR][CI] Automate nightly Delta known-failures updates #12855 opens or updates a deletion-only PR from the scheduled run, covering core-library fixes such as [GLUTEN-10134][CI] Remove Delta known-failures fixed by cast mode preservation #12815 and [MINOR][CI] Remove now-passing Delta type widening tests from known failures #12851 that did not trigger Delta CI when merged.. Setting to allow Github Actions to create PR is not enabled
• Move compiler and dependency caches to Apache Stash. Migrated ccache, Maven, and sbt/Ivy/Coursier caches in [MINOR][CI] Move Delta Spark UT caches to Apache Stash #12820.
• [GLUTEN-13037][CI] Extract temporary Delta test patches #13057 — Extract temporary Delta test patches into apply-delta-test-patches.sh, called by setup-delta.sh. The upstream scan cherry-picks, deterministic row-group fixtures, and 2B-row DV fail-fast patches now live there.
• Cache and reuse the fully built Velox native library across workflows (not ccache). gluten-delta/** and backends-velox/src-delta*/** match both velox_backend_x86.yml and delta_spark_ut.yml, so those PRs build the native lib twice (~10 min). Suggested by @philo-he and @zhouyuan. A prebuilt-library cache is preferable to artifact-passing: artifacts are scoped to a workflow run and there is no cross-workflow needs, so the Delta workflow would have to poll a sibling run it races with, whereas a prebuilt-library cache populated by push-to-main has no ordering problem. The key must cover cpp/**, the build flags and the base image — unlike ccache, where a bad key only costs a rebuild, a stale hit on a prebuilt lib means silently testing the wrong binary. A miss must fall back to building.
• Cover more Spark versions. The workflow already takes spark_version / delta_ref inputs, and pom.xml pairs Spark 3.5→Delta 3.3.2, 4.0→4.0.1, 4.1→4.1.0. The blocker is the baseline: each combination needs its own known-failures.txt and a bootstrap run. Suggested starting point: Spark 3.5 as a nightly-only second leg, which keeps per-PR cost unchanged. Requested by @zhouyuan.
• Remove the overlapping imported Delta suites. @malinjawi identified 8 suites imported from delta-io/delta (DeltaSuite, DeleteSQLSuite + DV variants, UpdateSQLSuite, DeltaDDLSuite, DeltaInsertIntoTableSuite, ClusteredTableClusteringSuite, OptimizeMetadataOnlyDeltaQuerySuite, OptimizedWritesSuite) that this pipeline now duplicates — but only on the Spark 4.1 leg. They are the only Delta coverage on Spark 3.5 and 4.0, so this is blocked on the Spark 3.5 leg above. Raised by @zhztheplayer.
• Delta 4.3.0 support. Attempted during [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline #12388: the bundle is compiled against Delta 4.1.0 and hits a binary-incompatible change (IdentityColumn.logTableWrite, first param Snapshot → SnapshotDescriptor), which NoSuchMethodErrors on every write. Needs the bundle built against 4.3.0.
• /delta-test PR comment opt-in. Replaces the proposed run-delta-ci label: contributors cannot apply labels, but anyone can comment /delta-test to force the full suite on a PR skipped by the paths: filter. Implemented in [GLUTEN-12743][CI] Let contributors trigger the Delta Spark UT with a /delta-test PR comment #12781.
• Drop the upstream cherry-picks ([Spark] [Test] Collect scans by FileSourceScanLike in ScanReportHelper delta-io/delta#7104 + [Spark] [Test] Match FileSourceScanLike instead of FileSourceScanExec delta-io/delta#7105, FileSourceScanExec → FileSourceScanLike) once DELTA_REF moves to a release that contains them. apply-delta-test-patches.sh already skips a cherry-pick that is already applied, so this is cleanup.

Notes for whoever picks these up

• Refresh the baseline by running Delta Spark UT (Gluten) with update_baseline=true and committing the produced artifact; see .github/workflows/util/delta-spark-ut/README.md. - A test that fails consistently belongs in known-failures.txt. flaky-tests.txt / flaky-error-patterns.txt are only for tests that genuinely pass on some runs and fail on others — quarantining a deterministic failure hides a real bug. - The nightly run enforces fail_on_fixed=true, so when a fix lands, the nightly goes red until the now-passing tests are removed from the baseline. That is the intended signal.