fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values

[peterxcli]

Rationale for this change

Spark's vectorized Parquet reader converts TIMESTAMP_MILLIS values with the checked millisToMicros (Math.multiplyExact), so overflow throws independently of ANSI mode. Comet's reader used an unchecked v * 1000, silently wrapping the value.

Spark can also discard filtered-out values before timestamp conversion using statistics, dictionary, and row-level filters. Comet cannot mirror every pruning path, so checked conversion must not run on values Spark would never read.

What changes are included in this PR?

• Add a checked top-level Timestamp(Millisecond) -> Timestamp(Microsecond) conversion in parquet_convert_array, the path the native Parquet scan actually takes. Unfiltered overflow now raises an error instead of silently wrapping, matching Spark in every eval mode.
• Retain the existing safe cast for filtered scans. This preserves Spark-successful queries when pruning differs, including dictionary-only exclusion, timestamp IN lists above DataFusion's 20-value pruning cutoff, nested predicates, and schema-evolution casts on unrelated filter columns.
• Remove the predicate-domain rewrite. The filtered-scan fallback makes it unnecessary and avoids changing empty NOT IN NULL semantics.
• Keep nested timestamp fields on the existing safe conversion because DataFusion cannot prune nested-field predicates before conversion.
• Remove the unreachable millis-to-micros arm from array_with_timezone: Spark logical timestamps are always microseconds, and Parquet conversion now happens on the actual reader path.
• Reject a microsecond physical timestamp with a top-level millisecond Spark logical target during planning (CometCastColumnExpr::try_new), since Spark read schemas represent logical timestamps in microseconds.
• Use Arrow's zero-copy cast for Int32 -> Date32 in date_from_unix_date, with consistent DEFAULT_CAST_OPTIONS for scalar and array inputs.

Follow-up #5517 tracks surfacing overflow as Spark's ArithmeticException("long overflow") instead of a raw Arrow compute error. Follow-up #5553 tracks checked nested-field conversion once DataFusion supports nested pruning.

How are these changes tested?

• make core
• DYLD_LIBRARY_PATH=$JAVA_HOME/lib/server cargo test -p datafusion-comet test_millis_to_micros_overflow_checked_only_at_top_level --lib — 1 passed
• ./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite filtered TIMESTAMP_MILLIS" -Dscalastyle.skip=true — 2 tests passed in 1 suite
• ./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite TIMESTAMP_MILLIS overflow fails in native scan" -Dscalastyle.skip=true — 1 test passed in 1 suite
• ./mvnw spotless:apply

The Spark regressions cover direct and dictionary pages, positive/negative overflow, ANSI on/off, row-filter pushdown on/off, the 20/21-item IN boundary, dictionary-only exclusion, widened filter columns, nested predicates, and empty NOT IN with a NULL input.

[andygrove]

Thanks for working on this. Replacing the hand-written loops with Arrow kernels is a good direction, and I especially like that you spotted Arrow's timezone adjustment in cast_with_options and worked around it with a metadata-only relabel. The comment explaining why the relabel has to happen first is really helpful, and the new +07:00 test cases genuinely guard it, since without the relabel they would be off by seven hours.

I have a few things I would like to work through before this goes in.

1. Micros to millis: Arrow truncates, Spark floors

native/core/src/parquet/cast_column.rs

Arrow's timestamp downscale is plain integer division (time_array.unary(|o| o / divisor) in arrow-cast), so it truncates toward zero. Spark's SparkDateTimeUtils.microsToMillis is Math.floorDiv(micros, MICROS_PER_MILLIS), and there is a comment there specifically about pre-1970 timestamps needing that adjustment. So for -1_500_001 micros Spark produces -1501 and we produce -1500.

The old unary(|v| v / 1000) had the same behavior, so this is not something the PR introduces. My concern is that the new test asserts -1_500_001 -> -1500, which bakes the divergence into a test as though it were intended. Moving to Arrow's cast also takes away the easy fix, since the hand-written closure could have simply become v.div_euclid(1000).

Would it make sense to keep a small kernel for this one case so the floor semantics can be matched? If you would rather keep the Arrow cast, could we file an issue and reference it next to the negative assertion so the expected value does not read as deliberate?

2. CastOptions::default() is safe: true

native/spark-expr/src/utils.rs

Could the millis to micros cast use safe: false instead of CastOptions::default()? Arrow branches on that flag for the upscale path. With safe: true it takes unary_opt(|o| o.checked_mul(mul)), which allocates a fresh null buffer and checks every element. With safe: false it takes try_unary, which reuses the input null buffer. So the default is doing an extra pass per batch compared to the unary this replaces.

There is a behavior argument too. Spark's millisToMicros is Math.multiplyExact and throws on overflow, so raising an error is closer to Spark than silently producing NULL. It would also line up with DataFusion's DEFAULT_CAST_OPTIONS, which is what cast_column.rs uses in this same PR.

3. Inconsistent cast options in date_from_unix_date

native/spark-expr/src/datetime_funcs/date_from_unix_date.rs

The two branches end up with different options. The array path gets CastOptions::default(), which is safe: true, while scalar.cast_to(...) resolves to cast_to_with_options(target, &DEFAULT_CAST_OPTIONS), which is safe: false. Int32 -> Date32 goes through cast_reinterpret_arrays, so it cannot fail either way today. But if the Signature::exact(vec![Int32]) is ever widened, the two paths would quietly disagree, one nulling and one erroring. Worth making them match while it is cheap to do?

For what it is worth, I checked the two things in this file that looked riskiest and both are fine. Int32 -> Date32 stays zero-copy, so there is no regression versus the manual Date32Array::new. And dropping the explicit ScalarValue::Null arm is safe, because can_cast_types has (Null, _) => true and the cast returns new_null_array.

4. Test coverage in cast_column.rs

Both evaluate tests moved from Timestamp(ms, None) to Timestamp(ms, Some("+07:00")), and the three deleted unit tests were the ones covering target_tz = None. I think that leaves the no-timezone case uncovered, which is the branch where relabel_array early-returns because the types already match. Could one of these keep a None target?

It would also be good to have a case where the input array already carries a timezone, say Timestamp(us, Some("UTC")) to Timestamp(ms, Some("America/New_York")). relabel_array overwrites whatever timezone the input has, and "relabel, do not shift" is exactly the property the workaround is protecting, so having that pinned down would help.

A note, not a request

In cast_date_to_timestamp, Arrow's Date32 -> Timestamp(us) is a plain unary(|x| (x as i64) * MICROSECONDS_IN_DAY) that ignores the safe flag, so a large Date32 wraps silently. Spark's daysToMicros uses Math.multiplyExact. That is identical to the old (d as i64) * 86_400 * 1_000_000, so I am not asking for anything here. I only wanted to note it so it is not mistaken for something the Arrow cast fixed.

[peterxcli]

@andygrove thanks for the review!

1. Micros → millis rounding

Changed this one conversion back to a small custom kernel because Arrow truncates negative values toward zero, while Spark uses floor division.

• Before: -1_500_001 / 1_000 = -1500
• Now: -1_500_001.div_euclid(1_000) = -1501

2. Millis → micros cast options for array_with_timezone in utils.rs

Changed to use DEFAULT_CAST_OPTIONS.

I also added a regression assertion using i64::MAX to confirm overflow returns an error from spark: https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala#L969-L972

3. date_from_unix_date option consistency

Changed to use DEFAULT_CAST_OPTIONS.

4. cast_column.rs test coverage

Expanded the array test to cover all three relevant timezone layouts:

• No timezone → no timezone
• No timezone → +07:00
• UTC → America/New_York

[andygrove]

Thanks for the quick turnaround. All four points from the last round look addressed, and pinning the new test values to Spark's own DateTimeUtilsSuite example is a nice touch. I ran the touched tests locally and they pass, and I re-checked the claims against arrow-cast 58.4.0 and Spark master.

A couple of things I confirmed so nobody has to re-derive them. SparkDateTimeUtils.microsToMillis really is Math.floorDiv, with a comment about pre-1970 timestamps, so div_euclid is the right call. millisToMicros really is Math.multiplyExact, so safe: false in utils.rs is the Spark-faithful choice. And there is no performance regression anywhere: the millis-to-micros upscale reinterprets to Int64 zero-copy and then uses try_unary, which reuses the input null buffer, so it is the same single allocation the old unary did. Date32 -> Timestamp(us) is one unary plus a same-type cast that early-returns, and Int32 -> Date32 is still cast_reinterpret_arrays. I also grepped for other micros-to-millis sites that might share the truncation bug, and the only ones are the two this PR fixes.

One thing on packaging that I would like to sort out before this merges. The div_euclid change is a real behavior fix rather than a refactor, since any pre-1970 timestamp with a sub-second component now yields a different value than before. Our changelog is generated from PR titles, so as refactor: this lands with no signal that timestamp results changed. Could we retitle to fix:, or split the floor fix out so it gets its own entry? Either way it would help to have the divergence tracked in an issue we can link from the code comment.

The description needs a refresh too. The first bullet still says we use Arrow casts for the Parquet micros-to-millis conversion, and that is the one case that ended up keeping a hand-written kernel. It would also be good to mention that millis-to-micros overflow went from silently wrapping to raising an error, since that is user visible as well.

Nothing needed on docs. No serde or expression registration changed, so the compatibility pages and expressions.md stay as they are. date_from_unix_date already has good SQL test coverage including the Spark min and max date boundaries, and both of its branches are now consistent, so I have nothing to raise there.

The rest of my comments are inline.

[peterxcli]

@andygrove thanks for the review! review change is pushed. please take another look, thanks!

1. div_euclid is a behavior fix, not a refactor.

Retitled the PR. The final patch rejects that invalid read-schema pair instead of using div_euclid.

2. Refresh the PR description and mention overflow.

Updated it to describe the planning rejection and that millis→micros overflow now errors instead of wrapping.

3. Is micros→millis reachable from Spark or Iceberg?

No. Spark read schemas use microsecond logical timestamps. Construction now returns DataFusionError::Plan, with a Spark source link.

4. Keep scalar target-timezone coverage.

The pair is now rejected before scalar or array evaluation. The planning test covers timezone-free and timezone-bearing fields.

5. Use DEFAULT_CAST_OPTIONS at the third site.

Done. The timezone-free Date32 -> Timestamp path now uses DEFAULT_CAST_OPTIONS.

6. Explain why millis→micros always errors on overflow.

Added links to Spark’s Parquet call site and checked millisToMicros implementation.

[andygrove]

Thanks for the rework. Rejecting the pair outright is a better answer than either version of the division fix, and the ParquetSchemaConverter link makes the invariant checkable. CI is fully green, including Iceberg 1.11, which is the PR-gated Iceberg job, so the scan paths that matter here did get exercised.

A few things I confirmed independently so you do not have to re-derive them.

Your reachability claim is correct. to_arrow_datatype at native/core/src/execution/serde.rs:93-96 maps Spark timestamps to microseconds only, and the JVM side agrees at Utils.scala:157-160. IcebergScanExec builds its schema from the same convert_spark_types_to_arrow_schema at planner.rs:1651. A millisecond logical target genuinely cannot occur.

Deleting the three tests costs no coverage. types_differ_only_in_field_names returns false for a flat (Timestamp, Timestamp) pair, so relabel_array was never on the timestamp path, and its List, Map, and Struct coverage is intact.

temporal.rs and date_from_unix_date.rs are now consistent on DEFAULT_CAST_OPTIONS, and both look right to me.

Nothing needed on docs. No serde or expression registration changed, so the compatibility pages and expressions.md stay as they are.

The one thing I would like to sort out before this merges is whether the overflow fix actually reaches the Parquet reader. I do not think it does. Details inline.

[peterxcli]

@andygrove thanks for another round of review, addressed all of your review. please take another look. TIA!

The overflow fix may not reach the actual Parquet reader.

Moved the checked millis -> micros conversion into parquet_convert_array, the real Parquet reader path. It now uses try_unary with mul_checked(1_000), so overflow returns an error instead of null.

Please verify the fix end to end against Spark behavior.

Added a native Parquet scan regression covering TimestampType and TimestampNTZType, positive and negative overflow, dictionary on/off, and ANSI on/off. It verifies both Spark and Comet report overflow.

Is the millis→micros arm in array_with_timezone reachable?

Confirmed it is unreachable for Parquet reads. Removed the dead arm, its imports, and its unit test. The conversion is now tested where it actually runs.

The millisecond-target rejection only checks top-level timestamps.

Narrowed the comment to explicitly state that the guard applies to top-level timestamp columns, so it does not imply nested timestamp validation.

[andygrove]

Note on this review: this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.

Turning the silent v * 1000 wrap into a checked multiply is the important part here, and linking to ParquetVectorUpdaterFactory and SparkDateTimeUtils.millisToMicros in the comment makes it easy to verify against Spark. Deleting the duplicate cast_timestamp_micros_to_millis_* helpers is a good cleanup.

Several things.

This conflicts directly with #5457

#5457 rewrites cast_date_to_timestamp in temporal.rs as well, replacing the chrono region-zone path with fixed-offset arithmetic and a plan-time fallback. This PR keeps that path and adds an Arrow fast path for the NTZ case. They cannot both land as written. Worth coordinating with @sunchao on which goes first.

date_from_unix_date loses a zero-copy path

The old code was:

Date32Array::new(int_array.values().clone(), int_array.nulls().cloned())

which is O(1): cloning an Arrow Buffer is a refcount bump. The new code calls cast_with_options(arr, &DataType::Date32, ...). Depending on the Arrow version, Int32 -> Date32 may or may not take a reinterpret fast path. If it does not, this turns a free operation into a full array copy on every batch.

Could you confirm which it is? If Arrow copies, the old code was better and I would keep it. The general principle of delegating to Arrow is right, but not when the hand-written version is asymptotically cheaper.

Separately, the new version accepts any type Arrow can cast to Date32, where the old one rejected anything that was not Int32. Spark's DateFromUnixDate only accepts IntegerType, so this should not matter, but it does remove a guard against a planner bug. Worth a comment saying the input type is guaranteed by the serde.

Removing the millis-to-micros arm from array_with_timezone

That arm is gone, so the same conversion in array_with_timezone now falls to Err("Not supported"). array_with_timezone is called from cast_array, not just the Parquet reader. Is there any path where a Comet CAST sees a Timestamp(Millisecond, None) input, for example from an Iceberg scan or a UDF return value? If there is, this changes a working conversion into an error. If there is not, it would be good to say so in the commit message, since the deletion looks unrelated to the Parquet fix at first glance.

What does the new overflow error look like to a user?

try_unary(... mul_checked ...) produces an ArrowError, which as far as I know surfaces as CometNativeException rather than the ArithmeticException("long overflow") Spark's Math.multiplyExact throws. #5169 is doing exactly this kind of error-fidelity work for the decimal paths. Is it worth making this one a typed SparkError from the start, rather than adding it to the list of raw Arrow errors that need converting later?

try_new returning a plan error

The new microsecond-physical to millisecond-target check returns DataFusionError::Plan from inside the schema adapter. If a file ever does present that shape, the query fails rather than falling back to Spark. The comment argues Spark's schema converter makes it unreachable, and the Spark link supports that. Could the message say what a user should do if they somehow hit it, or would it be better to fall back rather than fail?

[peterxcli]

@andygrove thanks for the review! Addressed three points and pushing back on two, details below.

This conflicts directly with #5457

Good catch. Since #5457 rewrites cast_date_to_timestamp as a safety fix and my change there was behavior-neutral (Arrow's NTZ arm compiles to the same unchecked days * MICROS_PER_DAY), I dropped the temporal.rs hunk from this PR entirely. #5457 can own that function; this PR now touches only the Parquet overflow paths, which also fits the fix: title better.

date_from_unix_date loses a zero-copy path

Confirmed on arrow-cast 58.4.0: (Int32, Date32) takes cast_reinterpret_arrays::<Int32Type, Date32Type> (mod.rs#L1631), which is a zero-copy buffer reinterpret — same cost as the old Date32Array::new, no regression. (Also matches your July 31 review note.) Added a comment documenting that the input is guaranteed Int32 by Signature::exact and the serde.

Removing the millis-to-micros arm from array_with_timezone

This was settled in the previous round — the arm was removed at your suggestion after we verified it unreachable: Spark logical timestamps are exclusively microseconds (serde.rs:95-98, Utils.scala:157-160), and both Iceberg scans and UDF return types flow through the same schema conversion, so cast_array can never see a millisecond input. I updated the PR description to note the rationale so the deletion doesn't look unrelated in the squashed commit.

What does the new overflow error look like to a user?

I'd rather not convert it in this PR. Spark's exception here is an untyped java.lang.ArithmeticException("long overflow") from Math.multiplyExact — there is no Spark error class to map to — and reusing SparkError::ArithmeticOverflow would emit "set spark.sql.ansi.enabled to false to bypass this error", which is actively misleading since this path throws regardless of ANSI (the new test pins that with ANSI both on and off). Proper fidelity needs a new native variant plus ShimSparkErrorConverter mappings per Spark version, which belongs with the #5169 error-fidelity effort. Filed #5517 to track it.

try_new returning a plan error

Fallback isn't reachable from there — by the time the native schema adapter runs, the JVM has already committed to the native scan. I extended the message to state it indicates a Comet bug, link the issue tracker, and suggest spark.comet.scan.enabled=false as a workaround.

[sunchao]

Worth coordinating with @sunchao on which goes first.

Feel free to merge this first!

[sunchao]

Reviewed afa3673 with five specialist agents; no blocking correctness findings. The inline dictionary fixture note does not block approval.

Local validation passed: 82 native Parquet tests, 643 native expression tests, and the Spark 4.1.3 overflow regression. A temporary 16-row variant also passed after verifying dictionary pages for both timestamp types and both signs. Native validation excluded HDFS; the full Spark suite was not run locally.

[sunchao]

Re-reviewed 5c8de42 with five specialist agents. One additional regression is described inline; the earlier dictionary fixture issue is fixed.

Local validation: 82 native Parquet tests, 660 native expression tests, and the Spark 4.1.3 overflow regression passed. The additional filtered-read regression passes with base c067e4e and fails on this head. Native builds excluded HDFS; the full Spark suite was not rerun locally.

[sunchao]

Re-reviewed 917286124cf39cd0c5f6eed5a1a06676e9679f28 against base 215ab706aa9af074e589f400cd6843dee9938030. The checked conversion still makes some queries fail after Spark would prune the offending data, and the empty NOT IN rewrite can discard NULL rows. Four P2 findings are detailed inline: dictionary pruning, timestamp IN lists longer than 20 values, widened filter columns, and originally empty NOT IN.

The ordinary-comparison, short-IN, null-safe-equality, and nested-predicate cases discussed previously now pass all 64 comparison executions. The dictionary-overflow fixture also now verifies actual dictionary encoding.

Validation:

• Fresh native rebuild of the reviewed head, with HDFS disabled; independent review scopes covered predicate semantics, scan/planner integration, conversion/ownership/errors, Spark execution, and test coverage.
• Spark 4.1.3 / JDK 17: both PR-added TIMESTAMP_MILLIS tests passed. The additional 152-query matrix ran separately on the head and base: 11 mismatches on the head (8 overflow failures and 3 row-loss cases), none on the base.
• A separate 8-query dictionary-pruning matrix per revision confirmed another failure: Spark and the base return zero rows for a dictionary-excluded value, while the head overflows with row-filter pushdown disabled. Plain-encoding and row-filter controls were also checked.
• 26 conversion/compatibility component tests passed. Independent predicate checks also passed for ordinary comparisons, nonempty IN/NOT IN, timezone combinations, signed boundaries, and boolean composition; originally empty lists reproduced the NULL-semantic defect.
• Base comparisons reused an archived build of the exact base, verified by SHA-256. The production JVM/protocol sources match between these revisions, and the packaged native-library hashes were verified for the accepted runs.

Current CI checks: 62 successful, 9 skipped. Local validation did not cover the full Spark suite, other Spark versions, or HDFS.

[sunchao]

I found one source-traced regression in the latest revision. Details inline.