[EPIC] Timezone handling bugs

[andygrove]

What / Why

I audited how Comet handles timezones, starting from #2730. The model is simple and mostly sound. Spark's TimestampType is a UTC instant, so nothing is converted at the JVM/native boundary. Comet passes the raw microseconds in both directions and labels them Timestamp(Microsecond, "UTC"). TimestampNTZType is Timestamp(Microsecond, None). The session timezone never becomes part of a value. Each timezone-aware expression carries it, and it's applied inside the native kernel, or the expression runs through the codegen dispatcher with Spark's own timeZoneId. #6337 adds a contributor guide page that describes the model in more detail.

The bugs cluster where that model breaks down:

• native expressions that emit a TimestampType value with some other label
• session timezone IDs that the native parser can't read
• timezone rules that come from a different database than the JVM's

Label drift is easy to miss. The scan and shuffle boundaries cast every column back to its declared type, so a test that only projects the result passes. It shows up when the result is compared, goes through a CASE, or feeds another native expression.

All of the new bugs below reproduce on main at 764936187, on Spark 3.5 and 4.1.

Bugs

• Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so hour, casts and comparisons over it are wrong or fail in non-UTC sessions (critical). Fixed by fix: label native timestamp_seconds results as UTC timestamps #6341.
• Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions, the default on Ubuntu and Debian images (high). Fixed by fix: keep the input's timezone label on native date_trunc results #6345.
• CASE and COALESCE over timestamps panic when the branches have different Arrow timezones #6327 CASE and COALESCE over timestamps panic when the branches have different Arrow timezones (high). Fixed by fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347.
• Session timezone IDs such as GMT+8, Z and PST make native timestamp expressions fail #6329 session timezone IDs such as GMT+8, Z and PST make casts, hour and df.show() fail (high). Fixed by fix: normalize session timezone IDs before passing them to native code #6351.
• timestamp_trunc panics on DST-transition timestamps in a DST timezone #5633 timestamp_trunc panics on DST-transition timestamps in a DST timezone. Native only with allowIncompatible (high). feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 fixes literal formats, and fix: truncate timestamps around DST transitions the way Spark does #6354 will cover per-row formats after it lands.
• Native timezone rules come from chrono-tz's bundled tzdata, so results differ from Spark when the JVM's tzdata differs #6331 native timezone rules come from chrono-tz's bundled tzdata (2025b), not the JVM's (medium). fix: warn when native and JVM timezone databases differ #6353 logs a warning when the two versions differ.
• Native CSV V2 scan parses timestamps as UTC and ignores the session timezone #6332 the native CSV V2 scan ignores the session timezone (low, testing-only config). Fixed by fix: fall back from the native CSV scan for timestamps outside UTC #6349.
• days transform is evaluated in the session timezone, while hours and Iceberg use UTC #6333 days is evaluated in the session timezone, while hours and Iceberg use UTC (low). fix: count the days partition transform in UTC #6348 is approved, and CI, including the Iceberg suites, is running.

Status (2026-10-05)

Six of the eight bugs are fixed on main. #6351 also replaced every getOrElse("UTC") fallback with CometTimeZone.nativeId, and #6347 turned the asserts in array_with_timezone into errors, so the section on #2730 below describes the code before those changes.

Still open besides #5633 and #6333:

• docs: update the timezone page for the fixes from the timezone EPIC #6686 brings the contributor guide page from docs: add contributor guide page on timezone handling #6337 up to date with fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347, fix: normalize session timezone IDs before passing them to native code #6351, fix: warn when native and JVM timezone databases differ #6353 and fix: fall back from the native CSV scan for timestamps outside UTC #6349. docs: check timezone handling against the contributor guide in the PR review skills #6340, which points the PR review skills at the page, is rebased and ready for review, and should merge after docs: update the timezone page for the fixes from the timezone EPIC #6686.
• There's still no systematic check that compares each native timestamp-returning expression with another timestamp, or feeds it to another native expression, in both Etc/UTC and a non-UTC session. timestamp_seconds (fix: label native timestamp_seconds results as UTC timestamps #6341) and date_trunc (fix: keep the input's timezone label on native date_trunc results #6345) got tests that compare their results, and session_timezone_ids.sql (fix: normalize session timezone IDs before passing them to native code #6351) covers the GMT+8-style IDs.

The UTC fallbacks in #2730

I instrumented every timeZoneId.getOrElse("UTC") site. Then I ran the datetime, cast, SQL-file, JSON, CSV, fuzz and expression suites. About 11,700 serde calls happened across 1,020 tests, and about 900 of them arrived without a timezone. Every one of those was a cast that Spark doesn't consider timezone-sensitive: numeric casts, Comet's own nullability-widening casts, and the cast inside IntegralDivide. None was a timezone-aware expression, which fits Spark refusing to resolve one without a timezone. So the fallback isn't a correctness bug today. The helper proposed in #6329 would replace it. The fallback can't simply be removed, though, because array_with_timezone asserts a non-empty timezone even for casts that don't use one.

Related

• Date-to-timestamp casts can overflow or panic for wide dates #5456 date-to-timestamp casts overflow or panic for wide dates (fix: make wide date-to-timestamp casts safe #5457 is open)
• Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code #5010 datetime rebasing isn't supported (feat: add native Parquet datetime rebasing #5047 and fix: fall back for Parquet datetime rebasing #5048 are open). Spark's legacy rebase is itself timezone-dependent.
• Evaluate moving from chrono to Jiff #4754 moving from chrono to jiff
• DataFusion / DataFusion-Spark functions whose Arrow return type drifts from Spark catalyst's declared type #4515 return type drift, where Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 and Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 are the timestamp cases with consequences inside a stage
• Audit Spark SQL configs that affect query semantics and ensure Comet honors them (or falls back) #4180 has an item for honoring spark.sql.session.timeZone everywhere

Already documented: Python Arrow UDFs see timestamps labelled UTC rather than the session timezone, and chrono-tz's DST rules end around 2100. Not in the user guide yet: spark.sql.parquet.int96TimestampConversion=true disables Comet for the session.

Fixed earlier, same class: #2720 (SparkToColumnar labelled timestamps with the session timezone), #2649 via #4761 (the date_trunc schema mismatch, whose fix introduced the label in #6330), and #5556 (the Python runner accepts Etc/UTC for UTC).

Test gaps

The SQL-file tests use UTC, America/Los_Angeles, America/New_York, Asia/Kolkata and +05:30. None of them use Etc/UTC, or the offset and short-ID forms from #6329. Most expression tests only project their result. Adding GMT+8 to the datetime files' ConfigMatrix would have caught #6329. #6328, #6330 and #6327 need more than a timezone setting: a test that compares each native timestamp-returning expression with another timestamp, or uses it in a CASE, in both a non-UTC session and Etc/UTC.