Native timezone rules come from chrono-tz's bundled tzdata, so results differ from Spark when the JVM's tzdata differs

[andygrove]

Describe the bug

Spark converts between instants and local time using the JVM's timezone rules (tzdb.dat), whose version depends on the JDK build and on whether TZUpdater has been run. Comet's native kernels use chrono-tz instead, which compiles its own copy of the IANA database into libcomet. chrono-tz 0.10.4 carries 2025b. When the two versions disagree about a zone, every local-time operation on the affected timestamps gives a different answer, and nothing reports it. That covers hour, casts to string and date, date_trunc, parsing strings into timestamps, and date-to-timestamp casts.

The drift can go either way. A JDK older than 2025b disagrees with Comet about recent rule changes. A JDK newer than 2025b will disagree about whatever changes after it, until we upgrade chrono-tz.

Steps to reproduce

With AppleJDK 17.0.10, which ships tzdata 2023c, on main at 764936187:

CREATE TABLE py USING parquet AS SELECT * FROM VALUES (TIMESTAMP'2025-06-15T12:00:00Z') AS v(ts);
CREATE TABLE kz USING parquet AS SELECT * FROM VALUES (TIMESTAMP'2024-06-30T23:30:00Z') AS v(ts);

SET spark.sql.session.timeZone=America/Asuncion;
SELECT hour(ts), CAST(ts AS STRING) FROM py;

SET spark.sql.session.timeZone=Asia/Almaty;
SELECT hour(ts), CAST(ts AS STRING) FROM kz;

For Asuncion, Spark returns 8, 2025-06-15 08:00:00 and Comet returns 9, 2025-06-15 09:00:00. For Almaty, Spark returns 5, 2024-07-01 05:30:00 and Comet returns 4, 2024-07-01 04:30:00. Paraguay moved to permanent UTC-3 (tzdata 2025a), and Kazakhstan unified on UTC+5 on 2024-03-01 (tzdata 2024a). 2023c has neither change.

Expected behavior

The same local times as Spark. Failing that, a documented limitation and a warning when the two versions differ.

Additional context

We could expose chrono_tz::IANA_TZDB_VERSION over JNI and log a warning at startup when it differs from the JVM's version (ZoneRulesProvider.getVersions("UTC").lastKey()). We could also document the drift next to the existing chrono-tz horizon note in the datetime compatibility guide, and keep chrono-tz current. #4754 is related, though jiff reading the system tz database wouldn't match the JVM's bundled copy either.