[GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics by felipepessoto · Pull Request #12967 · apache/gluten

felipepessoto · 2026-09-04T01:37:07Z

What changes are proposed in this pull request?

Enable native min/max aggregation for Spark TIMESTAMP_NTZ in the Velox backend, including the task-local SortAggregateExec used to collect Delta write statistics.

The implementation reuses Velox's existing Spark timestamp min/max kernels rather than introducing new aggregate kernels. TIMESTAMP_NTZ remains represented by the distinct Velox TIMESTAMP_UTC logical type, with microsecond precision (Velox implementation, Spark registration).

Accept TimestampNTZType in aggregate buffer/result and grouping type checks, and allow aggregate, shuffle, and the projection expressions needed by Delta statistics through the coarse NTZ fallback validator. Existing native function validation remains in place.
Use tsntz as the internal function-signature token. The previous ts_ntz token collides with _, which the native signature parser uses to separate argument types (parser).
Preserve TIMESTAMP_UTC as Substrait PrecisionTimestamp with precision 6 during type round-tripping.
Remove the 42 now-passing TIMESTAMP_NTZ data-skipping cases from the Delta known-failure baseline.

This addresses the NTZ aggregation failure encountered by the Delta statistics tracker in #12538. It does not implement a general fallback for arbitrary unsupported statistics plans or claim support for every NTZ expression. DATE/TIMESTAMP_NTZ cast support is handled separately in #12966.

Related #11622

How was this patch tested?

C++ signature, type, and min/max plan round-trip tests verify distinct NTZ type identity, Substrait precision 6, and preservation of microsecond values using Spark min/max.
GlutenTimestampNtzAggregateSuite on Spark 4.1 checks native SQL min/max execution and microsecond results, plus Spark fallback for NTZ JSON serialization in a non-UTC session timezone.
GlutenDeltaStatsSuite on Spark 3.5/Delta 3.3 and Spark 4.1/Delta 4.x checks writes and read-back of top-level and nested NTZ columns, with exact JSON-path assertions for their min/max statistics.
Existing DateFunctionsValidateSuite coverage exercises NTZ scalar functions after the signature-token change.
The Delta Spark UT workflow confirms the 42 data-skipping cases pass; unrelated type-widening failures remain in the baseline.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.83

github-actions · 2026-09-04T01:37:36Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T01:37:37Z

🔄 Delta Spark UT started by @felipepessoto (~2.5 h). View run

github-actions · 2026-09-04T01:39:45Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T01:49:16Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T02:49:25Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T02:54:45Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T04:58:16Z

Run Gluten Clickhouse CI on x86

Copilot

🟡 Changes recommended

A newly added Velox round-trip test uses millisecond-precision timestamps despite declaring microsecond precision, weakening coverage for the intended NTZ precision behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables native handling of TIMESTAMP_NTZ (Spark) / TIMESTAMP_UTC (Velox) in aggregation and related Delta statistics collection paths for the Velox backend, reducing Spark fallbacks and unblocking Delta stats plans previously rejected due to NTZ types.

Changes:

Extend the TimestampNTZ fallback validator to allow aggregates, shuffles, and direct NTZ projections needed by Delta stats plans.
Accept TimestampNTZType in aggregation buffer type checks and preserve a distinct native signature token (tsntz) plus Substrait PrecisionTimestamp encoding.
Add native regression coverage (Spark UT + Velox C++ tests) and remove now-fixed Delta data-skipping cases from the known-failure baseline.

File summaries

File	Description
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala	Adds Spark 4.1 regression coverage for NTZ min/max aggregation and a projection fallback case.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Enables the new Spark 4.1 UT suite in Velox test settings.
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala	Broadens NTZ fallback validation to permit additional plan nodes/expressions used by native stats aggregation.
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala	Maps Spark `TimestampNTZType` to Substrait timestamp NTZ type node and uses `tsntz` in signature naming.
gluten-substrait/src/main/scala/org/apache/gluten/execution/HashAggregateExecBaseTransformer.scala	Allows `TimestampNTZType` in supported aggregation buffer type checks.
cpp/velox/tests/VeloxToSubstraitTypeTest.cc	Adds a type-conversion test for `TIMESTAMP_UTC` -> Substrait `PrecisionTimestamp`.
cpp/velox/tests/VeloxSubstraitSignatureTest.cc	Adds `tsntz` signature coverage for `TIMESTAMP_UTC` mapping in both directions.
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc	Adds a min/max aggregation round-trip test over `TIMESTAMP_UTC`.
cpp/velox/substrait/VeloxToSubstraitType.cc	Encodes `TIMESTAMP_UTC` as Substrait `precision_timestamp(6)`.
cpp/velox/substrait/VeloxSubstraitSignature.cc	Maps `TIMESTAMP_UTC` to/from the `tsntz` signature token.
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 4.0 stats regression coverage including nested NTZ columns.
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 3.3 stats regression coverage including nested NTZ columns.
.github/workflows/util/delta-spark-ut/known-failures.txt	Removes Delta data-skipping known failures resolved by native NTZ stats aggregation support.

Review details

Files reviewed: 13/13 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

felipepessoto · 2026-09-04T09:44:37Z

@Mariamalmesfer @rui-mo I think you two have worked on other timestamp_ntz PRs. Could you take a look, please?

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

github-actions · 2026-09-04T18:38:07Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T19:20:51Z

Run Gluten Clickhouse CI on x86

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:1

The previous implementation used dataType.typeName == "timestamp_ntz", which is resilient across Spark versions/shims. Switching to a direct TimestampNTZType reference introduces a compile-time dependency that can break builds for Spark variants where TimestampNTZType is absent or shaded differently. If this module is cross-built across multiple Spark versions, consider keeping a version-tolerant check (e.g., match TimestampNTZType when available and fall back to typeName), ideally via a shim utility.

/*

github-actions · 2026-09-04T20:13:53Z

Run Gluten Clickhouse CI on x86

Copilot

🟢 Approval recommended

The changes are cohesive and well-covered by targeted Spark + native regression tests, and the updated validator/signature/type mappings are consistent across JVM and C++ paths.

Review details

Files reviewed: 13/13 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 · 3 · 2

Open (7)

toSubstraitType unconditionally sets PrecisionTimestamp nullability to NULLABLE for… · New
Nullability is being hard-coded to NULLABILITY_NULLABLE for TIMESTAMP_UTC. This can…
This test validates precision and type equivalence but doesn’t assert nullability behavior. Add… · New
This assertion is brittle because it requires every BaseAggregateExec in the executed plan to…
This introduces a JVM-global, mutable observer that can be accessed from multiple task threads.…
scala.collection.JavaConverters._ is deprecated on Scala versions that provide… · New
The Delta 3.3 and Delta 4.0 versions of GlutenDeltaStatsSuite appear to be identical in this PR.…

felipepessoto · 2026-09-22T23:06:42Z

+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitPrecisionTimestamp);
+    return *substraitType;
+  }


This converter currently accepts only (arena, type) , with no nullability argument. The parser does not retain nullability in the Velox type, and existing conversion branches—including ordinary TIMESTAMP —also emit NULLABLE . Local probes confirmed that required inputs become nullable for both timestamp types, so this behavior is not introduced by the NTZ branch. @rui-mo, should we open a separate issue to track preserving schema nullability end to end, since this is existing converter-wide behavior?

github-actions · 2026-09-22T18:30:10Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Observe the task-local statistics plan in tests and require native aggregate offload in both Delta variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add grouped min/max coverage with TIMESTAMP_NTZ keys and values. Check exact per-key results and require every aggregate stage to use native execution with an NTZ grouping key. Clarify that disabling TimestampNTZ validation permits supported scans, aggregates, shuffles and projections while other validation rules still apply. Regenerate the public configuration reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Describe the high-level fallback behavior of enableTimestampNtzValidation without enumerating the evolving set of supported operators and expressions. Keep the other-validation caveat and regenerate the public configuration reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Allow IsNotNull alongside IsNull using the existing NTZ projection allowlist pattern. Add coverage for both null predicates and a compound projection, checking results and native execution when validation is disabled while preserving Spark fallback when it is enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-25T22:42:29Z

Run Gluten Clickhouse CI on x86

Copilot

Copilot review overview

🔵 Needs a closer look

The cross-language type, signature, aggregation, and Delta changes require final human review.

Review effort: Lite
Findings: 1

Open (1)

toSubstraitType unconditionally sets PrecisionTimestamp nullability to NULLABLE for…

Resolved since last review (6)

Nullability is being hard-coded to NULLABILITY_NULLABLE for TIMESTAMP_UTC. This can…
This test validates precision and type equivalence but doesn’t assert nullability behavior. Add…
This assertion is brittle because it requires every BaseAggregateExec in the executed plan to…
This introduces a JVM-global, mutable observer that can be accessed from multiple task threads.…
scala.collection.JavaConverters._ is deprecated on Scala versions that provide…
The Delta 3.3 and Delta 4.0 versions of GlutenDeltaStatsSuite appear to be identical in this PR.…

…ng MERGE's parents on Spark, 42 already gone on main The delta_spark_ut run on 45bd4ad reports zero regressions and these now-passing entries: - 40 ImplicitMergeCastingSuite "MERGE overflow in WHEN MATCHED THEN UPDATE" cases. With only the guard, Spark's write projection raised SparkArithmeticException [CAST_OVERFLOW] as expected, but the RowToVeloxColumnar feeding the native filter above it wrapped that in a GlutenException, so the tests saw the wrong exception type. Keeping the filter and column drop on Spark removes the transition and the original exception propagates. - 42 DataSkippingDeltaV1* TIMESTAMP_NTZ cases, which apache#12967 fixed and removed from the baseline on main after this branch was cut; CI tests the merge with main, so they only show against this branch's copy of the file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Preserve NTZ logical types and microsecond values across JVM, Substrait and Velox literal conversion. Distinguish timestamp from timestamp_tz, retain complete types for null literals, and handle scalar vector-backed constants without a virtual-table container. Connect the existing JVM NTZ literal node to the literal factory and nested type handling. Include the NTZ type mapping required by independent literal and Values round trips, with dedicated native and Spark 4.1 regression coverage. Follow-up to apache#12967 (comment) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Spark `to_json` currently applies the session timezone to `TIMESTAMP_UTC` (`TIMESTAMP_NTZ`) values and appends an offset. For example, in an `America/Los_Angeles` session: ```sql SELECT to_json(named_struct('ts', TIMESTAMP_NTZ '2024-01-01 00:00:00.123456')); ``` ```text Before: {"ts":"2023-12-31T16:00:00.123-08:00"} After: {"ts":"2024-01-01T00:00:00.123"} ``` Use timezone-free formatting for NTZ values, including values nested in collections. Ordinary timestamp formatting and timestamp map-key encoding are unchanged. This follows [Spark's NTZ JSON formatting](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala#L147-L156) and addresses serializer behavior exposed while reviewing apache/gluten#12967. Generated-by: GitHub Copilot CLI 1.0.83 Differential Revision: D125968265 Pull Request resolved: #19199

Preserve NTZ logical types and microsecond values across JVM, Substrait and Velox literal conversion. Distinguish timestamp from timestamp_tz, retain complete types for null literals, and handle scalar vector-backed constants without a virtual-table container. Connect the existing JVM NTZ literal node to the literal factory and nested type handling. Include the NTZ type mapping required by independent literal and Values round trips, with dedicated native and Spark 4.1 regression coverage. Follow-up to apache#12967 (comment) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added CORE works for Gluten Core VELOX labels Sep 4, 2026

felipepessoto closed this Sep 4, 2026

felipepessoto reopened this Sep 4, 2026

github-actions Bot added the INFRA label Sep 4, 2026

felipepessoto marked this pull request as ready for review September 4, 2026 09:33

Copilot AI lite review requested due to automatic review settings September 4, 2026 09:33

Copilot started reviewing on behalf of felipepessoto September 4, 2026 09:39

View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/velox/tests/VeloxSubstraitRoundTripTest.cc Outdated

Copilot AI review requested due to automatic review settings September 4, 2026 17:57

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot started reviewing on behalf of felipepessoto September 4, 2026 18:38

View session

felipepessoto mentioned this pull request Sep 4, 2026

[GLUTEN-12538][VL] Fall back to row-based Delta stats when the stats plan cannot be offloaded #12539

Closed

Copilot AI review requested due to automatic review settings September 4, 2026 19:06

Copilot started reviewing on behalf of felipepessoto September 4, 2026 19:21

View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/velox/substrait/VeloxSubstraitSignature.cc

Comment thread gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala

Copilot AI review requested due to automatic review settings September 4, 2026 20:13

Copilot started reviewing on behalf of felipepessoto September 4, 2026 20:13

View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Copilot started reviewing on behalf of felipepessoto September 22, 2026 18:30

View session

Copilot started reviewing on behalf of felipepessoto September 22, 2026 18:54

View session

felipepessoto mentioned this pull request Sep 24, 2026

fix(spark): Preserve TIMESTAMP_UTC values in to_json facebookincubator/velox#19199

Merged

felipepessoto and others added 12 commits September 25, 2026 22:41

[GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation

346ccf3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[MINOR] Document TIMESTAMP_NTZ signature token

b139394

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Fix NTZ aggregate validation coverage

90131c9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Update Delta test baseline

e906c6d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Restore unrelated Delta failures

f359a53

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Test microsecond NTZ round trip

6d51672

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Address follow-up review comments

da731a9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Localize NTZ fallback assertion

7a4bf75

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Assert native Delta statistics aggregation

81ad0ef

Observe the task-local statistics plan in tests and require native aggregate offload in both Delta variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings September 25, 2026 22:41

felipepessoto force-pushed the gluten-12538-timestamp-ntz-sort-aggregate branch from 8f6587c to 92e2698 Compare September 25, 2026 22:41

Copilot started reviewing on behalf of felipepessoto September 25, 2026 22:42

View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

zhouyuan merged commit 321229a into apache:main Sep 29, 2026
71 checks passed

felipepessoto deleted the gluten-12538-timestamp-ntz-sort-aggregate branch September 29, 2026 11:59

malinjawi mentioned this pull request Sep 30, 2026

[GLUTEN-9003][DELTA] Keep conditional IncrementMetric on Spark #13158

Closed

felipepessoto mentioned this pull request Oct 15, 2026

[Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline #12743

Open

7 of 21 tasks

Conversation

felipepessoto commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

felipepessoto commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

felipepessoto Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

felipepessoto commented Sep 4, 2026 •

edited

Loading